Key Points

References

Key Concepts

More test tools

testing tools

https://github.com/TestLinkOpenSourceTRMS/testlink-code
https://www.google.com/search?client=firefox-b-1-d&q=testlink+tutorial
https://www.guru99.com/testlink-tutorial-complete-guide.html

https://jmeter.apache.org/
https://jmeter.apache.org/usermanual/index.html
https://jmeter.apache.org/usermanual/jmeter_proxy_step_by_step.html
https://cwiki.apache.org/confluence/display/JMETER/ProblemsRecording

https://www.selenium.dev/
https://www.selenium.dev/projects/

openapi

https://robotframework.org/
https://www.tutorialspoint.com/robot_framework/index.htm
https://github.com/robotframework/robotframework

Older tools that may not work

Canoo WebTest

https://github.com/canoo/webtest

https://www.testwo.com/blog/4811

canoo-webtest-review-2011-wo.com-Canoo webtest-vs-selenium-webtest-wins-13-5.pdf

curl

https://flaviocopes.com/http-curl/

https://curl.haxx.se/docs/manual.html

curl is a a command line tool that allows to transfer data across the network.

It supports lots of protocols out of the box, including HTTP, HTTPS, FTP, FTPS, SFTP, IMAP, SMTP, POP3, and many more.

When it comes to debugging network requests, curl is one of the best tools you can find.

It’s one of those tools that once you know how to use you always get back to. A programmer’s best friend.

It’s universal, it runs on Linux, Mac, Windows. Refer to the official installation guide to install it on your system.

Fun fact: the author and maintainer of curl, swedish, was awarded by the king of Sweden for the contributions that his work (curl and libcurl) did to the computing world.

Let’s dive into some of the commands and operations that you are most likely to want to perform when working with HTTP requests.

Those examples involve working with HTTP, the most popular protocol.

• Perform an HTTP GET request
• Get the HTTP response headers
• Only get the HTTP response headers
• Perform an HTTP POST request
• Perform an HTTP POST request sending JSON
• Perform an HTTP PUT request
• Follow a redirect
• Store the response to a file
• Using HTTP authentication
• Set a different User Agent
• Inspecting all the details of the request and the response
• Copying any browser network request to a curl command

Perform an HTTP GET request

When you perform a request, curl will return the body of the response:

curl https://flaviocopes.com/

Get the HTTP response headers

By default the response headers are hidden in the output of curl. To show them, use the i option:

curl -i https://flaviocopes.com/

Only get the HTTP response headers

Using the I option, you can get only the headers, and not the response body:

curl -I https://flaviocopes.com/

Perform an HTTP POST request

The X option lets you change the HTTP method used. By default, GET is used, and it’s the same as writing

curl -X GET https://flaviocopes.com/

Using -X POST will perform a POST request.

You can perform a POST request passing data URL encoded:

curl -d "option=value&something=anothervalue" -X POST https://flaviocopes.com/

Perform an HTTP POST request sending JSON

Instead of posting data URL-encoded, like in the example above, you might want to send JSON.

In this case you need to explicitly set the Content-Type header, by using the H option:

curl -d '{"option": "value", "something": "anothervalue"}' -H "Content-Type: application/json" -X POST https://flaviocopes.com/

You can also send a JSON file from your disk:

curl -d "@my-file.json" -X POST https://flaviocopes.com/

curl tutorial

curl-flaviocopes.com-The curl guide to HTTP requests.pdf

Swagger - OpenAPI

Postman - API Test Tool

Free Postman account

jm9@gmail

https://web.postman.co/workspaces?type=personal

api requests in a collection can be created, copied, moved, deleted

Create a public login request

/login in a collection in the workspace

create an API request test

url

{{authenticationBaseUrl}}/login

body

{
"email": "{{userEmail}}",
"password": "{{userPassword}}"
}

authorization type for request

headers

cookies ( 1 )

__cfduid=d788c04eea1b79504ecad092f49a88c4f1565225862; path=/; domain=.dmx.io; HttpOnly; Expires=Fri, 07 Aug 2020 00:57:42 GMT;

create test scripts

before and after request

after request script - sets environment variables from returned json jwt

const jsonData = JSON.parse(responseBody);
postman.setEnvironmentVariable("DMXToken", jsonData.token);
postman.setEnvironmentVariable("userId", jsonData.user._id);
const authDataUserEmail = jsonData.user.email;
postman.setEnvironmentVariable("userEmail", authDataUserEmail);
const authDataUserEmailDomain = authDataUserEmail.split("@")[1];
postman.setEnvironmentVariable("userEmailDomain", authDataUserEmailDomain);

How to Use Postman

https://www.getpostman.com/how-api-collaboration-works

Postman Errors at Runtime

Many runtime errors can be analyzed in the Postman client by:

1. validating the url for the host app is correct
2. ensuring the request url is correct with the substituted values for any values
3. viewing Postman console for the request / response data actually sent and received
4. viewing Postman logs for requests/ responses
5. checking headers, authentication methods on the test case
6. checking pre-request scripts
7. checking the test scripts for potential errors based on missing data in the response ( try removing the post tests temporarily and re-run the test )

Resolving an issue in the Deprecated Chrome Postman Plugin for Javascript support in web page accessed

https://github.com/postmanlabs/postman-app-support/issues/2646

Problem

Sending a request to some pages, keep getting the error 'Javascript is disabled on your browser. Please enable Javascript and refresh this page'. The page works on the browser but on Postman it comes up with that error. Initially thought it was the page so tried another and still keep getting the error

Solution

1. Postman Version: 4.9.3
2. App (Chrome app or Mac app): Mac app
3. OS details: 10.12.2
4. Is the Interceptor on and enabled in the app: No
5. Did you encounter this recently, or has this bug always been there: N/A
6. Expected behaviour: N/A
7. Console logs (http://blog.getpostman.com/2014/01/27/enabling-chrome-developer-tools-inside-postman/ for the Chrome App, View->Toggle Dev Tools for the Mac app): N/A
8. Screenshots (if applicable) N/A

For some reason, I need run javascript and submit forms in Preview. I know postman doesn't support it for security (CSP). So I inspected postman's source code via DevTools, and finally I found

131078                 { className: 'response-body-iframe-viewer' },
131079                 _react2.default.createElement('iframe', {
131080                   className: 'response-body-viewer-preview',
131081                   sandbox: '' // https://developer.mozilla.org/en/docs/Web/HTML/Element/iframe#attr-sandbox
131082                   , src: this.props.src
131083                 })

in /Applicatoins/Postman.app/Contents/Resources/app.js

Then I modify line 131081 to
sandbox: 'allow-forms allow-scripts'

relaunch postman, it works!

I think it's may not safe when using postman under sandbox: 'allow-forms allow-scripts' all the time. So maybe we could add a button to postman's preferences settings, then we can enable/disable it as needed.

Automating API tests in Postman

https://www.postman.com/use-cases/api-testing-automation

Robot Framework for Test Automation

is Robotframework's HttpLibrary (Requests)

https://robotframework.org/

Robot Framework is a generic open source automation framework. It can be used for test automation and robotic process automation (RPA).

Robot Framework is actively supported, with many industry-leading companies using it in their software development.

Robot Framework is open and extensible and can be integrated with virtually any other tool to create powerful and flexible automation solutions. Being open source also means that Robot Framework is free to use without licensing costs.

Robot Framework has easy syntax, utilizing human-readable keywords. Its capabilities can be extended by libraries implemented with Python or Java. The framework has a rich ecosystem around it, consisting of libraries and tools that are developed as separate projects.

Robot Framework project is hosted on GitHub where you can find further documentation, source code, and issue tracker. Downloads are hosted at PyPI.

Robot Framework is operating system and application independent. The core framework is implemented using Python and also runs on Jython (JVM) and IronPython (.NET).

Robot Framework itself is open source software released under Apache License 2.0, and most of the libraries and tools in the ecosystem are also open source

*** Settings ***
Documentation     A test suite with a single test for valid login.
...
...               This test has a workflow that is created using keywords in
...               the imported resource file.
Resource          resource.txt

*** Test Cases ***
Valid Login
    Open Browser To Login Page
    Input Username    demo
    Input Password    mode
    Submit Credentials
    Welcome Page Should Be Open
    [Teardown]    Close Browser

Run Reports

Define Keywords for Test Case ( variable ) 

*** Settings ***
Suite Setup       Open Browser To Login Page
Suite Teardown    Close Browser
Test Setup        Go To Login Page
Test Template     Login With Invalid Credentials Should Fail
Resource          resource.txt

*** Test Cases ***               User Name        Password
Invalid Username                 invalid          ${VALID PASSWORD}
Invalid Password                 ${VALID USER}    invalid
Invalid Username And Password    invalid          whatever
Empty Username                   ${EMPTY}         ${VALID PASSWORD}
Empty Password                   ${VALID USER}    ${EMPTY}
Empty Username And Password      ${EMPTY}         ${EMPTY}

*** Keywords ***
Login With Invalid Credentials Should Fail
    [Arguments]    ${username}    ${password}
    Input Username    ${username}
    Input Password    ${password}
    Submit Credentials
    Login Should Have Failed

Login Should Have Failed
    Location Should Be    ${ERROR URL}
    Title Should Be    Error Page
                    

Set Values for Keywords in Test

*** Settings ***
Library           SeleniumLibrary

*** Variables ***
${SERVER}         localhost:7272
${BROWSER}        Firefox
${DELAY}          0
${VALID USER}     demo
${VALID PASSWORD}    mode
${LOGIN URL}      http://${SERVER}/
${WELCOME URL}    http://${SERVER}/welcome.html
${ERROR URL}      http://${SERVER}/error.html

*** Keywords ***
Open Browser To Login Page
    Open Browser    ${LOGIN URL}    ${BROWSER}
    Maximize Browser Window
    Set Selenium Speed    ${DELAY}
    Login Page Should Be Open

Login Page Should Be Open
    Title Should Be    Login Page

Go To Login Page
    Go To    ${LOGIN URL}
    Login Page Should Be Open

Input Username
    [Arguments]    ${username}
    Input Text    username_field    ${username}

Input Password
    [Arguments]    ${password}
    Input Text    password_field    ${password}

Submit Credentials
    Click Button    login_button

Welcome Page Should Be Open
    Location Should Be    ${WELCOME URL}
    Title Should Be    Welcome Page

Libraries contain framework functions

Libraries provide the actual automation and testing capabilities to Robot Framework by providing keywords. Several standard libraries are bundled in with the framework, and galore of separately developed external libraries that can be installed based on your needs. Creating your own libraries is a breeze.

Standard

Builtin
Provides a set of often needed generic keywords. Always automatically available without imports.

Operating System
Enables various operating system related tasks to be performed in the system where Robot Framework is running.

String
Library for generating, modifying and verifying strings.

Process
Library for running processes in the system. New in Robot Framework 2.8.

External

Dialogs
Provides means for pausing the execution and getting input from users.

Remote
Special library acting as a proxy between Robot Framework and libraries elsewhere. Actual libraries can be running on different machines and be implemented using any programming language supporting XML-RPC protocol.

Telnet
Makes it possible to connect to Telnet servers and execute commands on the opened connections.

DateTime
Library for date and time conversions. New in Robot Framework 2.8.5.

Other

Collections
Provides a set of keywords for handling Python lists and dictionaries.

Screenshot
Provides keywords to capture screenshots of the desktop.

XML
Library for generating, modifying and verifying XML files.

Tools 

Built in Tools 

Editor Plugins for many Editors

Plugins for Build Tools 

Other RPA Automation Tools for Robot

Learn Postman

https://learning.getpostman.com/

Microshed - test framework for Java microservices in containers

https://github.com/MicroShed/microshed-testing

MicroShed Testing offers a fast and simple way of writing and running true-to-production integration tests for Java microservice applications. MicroShed Testing exercises your containerized application from outside the container so you are testing the exact same image that runs in production.

MicroShed Testing aims to:

1. be easy to get started with
2. work with any Java EE, Jakarta EE or MicroProfile runtime
3. provide true-to-production tests

Add Add microshed-testing-testcontainers and junit-jupiter as test-scoped dependencies to a Maven project

Run with Maven:

./gradlew publishToMavenLocal
cd sample-apps/maven-app
mvn clean install

Run with Gradle:

./gradlew :microshed-testing-jaxrs-json:test

NOTE: The first run will take longer due to downloading required container layers. Subsequent runs will be faster.

Runs with JEE containers:  TomEE, OpenLiberty etc

Sample Java-RS application

@Path("/people")
@ApplicationScoped
@Produces(MediaType.APPLICATION_JSON)
@Consumes(MediaType.APPLICATION_JSON)
public class PersonService {

    private final PersonRepo personRepo = // ...

    @GET
    public Collection<Person> getAllPeople() {
        return personRepo.values();
    }

    @GET
    @Path("/{personId}")
    public Person getPerson(@PathParam("personId") long id) {
        Person foundPerson = personRepo.get(id);
        if (foundPerson == null)
            throw new NotFoundException("Person with id " + id + " not found.");
        return foundPerson;
    }
    
    // ...
}

Microshed integration tests for Java - RS example

@MicroShedTest
public class BasicJAXRSServiceTest {

    // This will search for a Dockerfile in the repository and start up the application
    // in a Docker container, and wait for it to be ready before starting the tests.
    @Container
    public static MicroProfileApplication app = new MicroProfileApplication()
                    .withAppContextRoot("/myservice");

    // This injects a REST _Client_ proxy of the PersonService shown above
    // This allows us to easily invoke HTTP requests on the running application container
    @Inject
    public static PersonService personSvc;

    @Test
    public void testGetPerson() {
        // This invokes an HTTP POST request to the running container, which triggers
        // the PersonService#createPerson endpoint and returns the generated ID
        Long bobId = personSvc.createPerson("Bob", 24);
        
        // Using the generated ID, invoke an HTTP GET request to read the record we just created
        // The JSON response will be automatically converted to a 'Person' object using JSON-B 
        Person bob = personSvc.getPerson(bobId);
        
        assertEquals("Bob", bob.name);
        assertEquals(24, bob.age);
        assertNotNull(bob.id);
    }
    
    @Test
    public void testGetUnknownPerson() {
        // This invokes an HTTP GET request to get a person with ID -1, which does not exist
        // asserts that the application container returns an HTTP 404 (not found) exception
        assertThrows(NotFoundException.class, () -> personSvc.getPerson(-1L));
    }

    // ...
}

Potential Value Opportunities

Potential Challenges

Candidate Solutions

Webinars voice to Google docs text 

speech to text recognition 

requires:

Windows 10, Chrome, Google ChromeVox extension, Google Docs, Google Drive, CABLE-Input ( VB-Audio-Cable )

steps after setup done 

create new gdoc

select tools voice typing

test 

use the audio input from the speaker with the CABLE-INPUT device 

start the webinar presentation in another window

voice typing runs in the gdoc 

after webinar recording stops, format sentences and paragraphs using groovy script

optionally, download gdoc as a docx file if needed

https://www.howtogeek.com/364369/how-to-record-your-pc%E2%80%99s-audio-with-virtual-audio-cable/

API Test Tools List

An API or Application programming interface is a collection of software functions and procedures through which other software applications can be accessed or executed. In API Testing you use software to send calls to the API, get output and log the system's response. For Agile development, Api Testing becomes important as shorter development cycles put more pressure on automated testing.

Here is a list of top Web Services Testing Tools

1) SOAtest

Parasoft SOAtest is the industry-leading API testing solution, which simplifies the process of creating automated end-to-end test scenarios across multiple layers of modern applications (i.e. mobile, REST APIs, SOAP services, Microservices, databases, Web UIs, ESBs, or mainframes) from a single intuitive interface.

• Rapid and scriptless test creation: Automatically generate tests from a service definition to test design of an API. Use drag and drop tools for data driving, looping and performing complex assertions
• Save time building tests: SOAtest plug-in leverages artificial intelligence to automatically convert UI tests into scriptless API test scenarios
• Change management: SOAtest can identify service schema changes and automatically update your library of tests
• Maximum scalability: Integration for existing SDLC test frameworks and CI infrastructures. Seamless integration with load testing and service virtualization
• Robust and flexible integration: Broad support for 120+ protocols and message formats including JMS, MQ, TCP, File, Copybook, FIX, EDI and many more

2) API Fortress

API Fortress is the most powerful API testing and monitoring solution for REST and SOAP APIs.

• Comprehensive test generation with a single click.
• Web-based collaborative tool for teams. Works within your browser and requires no downloads.
• Intuitive UI that is easy to use at any skill level.
• Simple one-click automation. Test during development and deployments (CI/CD), and monitor functional uptime.
• Robust notification and data integrations.
• Works in the cloud, on-premises, or both in an ad hoc manner.

Hundreds of enterprises have already made the switch. More power, less confusion. The easiest way to automate your API testing and monitoring strategy.

3) Checkly

Checkly is a fresh new API monitoring and site transaction monitoring platform. With its great UI and dashboards, monitoring your crucial API endpoints and vital site click paths has never been easier.

• Flexible and deep API monitoring with support for GraphQL, XML, JSON and extensive request customisation.
• Puppeteer scripted browser clicks flows with assertions and screenshots.
• Email, SMS, Slack and Pagerduty alerting integrations.
• Great public, customisable dashboards & status pages for customers or in the office.
• Full scripting flexibility: use Node.js code to set up custom requests or reset test data.
• Easy CI/CD integration for ad hoc tests and validation.

4) APImetrics

APImetrics active API monitoring gives you up-to-date, accurate reporting of all the APIs and micro-services you supply or depend on, and how they impact your business.

• Test API collections in the real world, from real application servers, hosted on the cloud servers your clients use including AWS, Azure, Google and IBM
• Identify content errors, passing ‘failures’ and networking configuration problems
• State of the art AI trained on a database of over 1 billion API calls to identify hard to spot issues and take you straight to problems
• Integrated analysis and reporting including real time alerts, dashboards and quality scoring with Cloud API Service Consistency (CASC) Scores, a credit score for your API, benchmark your API to your industry

5) Ping API

Ping-API is API testing allows to write test script in JavaScript and CoffeeScript to test your APIs. It allows inspecting HTTP API call with a complete request and response data. For any failures, the user gets a notification through email, slack or Hipchat.

Features:

• Ping- API to schedule test in every minutes or hour
• Support for writing script to set request headers, body and URL parameters. It supports for writing script to validate response headers and body
• Validate CRUD flow and log in to Ping API

Download link: https://ping-api.com/

It provides an extensible framework helpful in executing and building the functionality of headless system that do not have a user interface. It helps to test the headless technologies like Databases and Webservices, JMS, etc. By using the API test conversion tool, you can convert soapUI tests to UFT (QTP) API tests.

Download QTP

vREST provides an online solution for automated testing, mocking, automated recording and specification of REST/HTTP APIs/RESTful APIs.

• It provides an exhaustive tool to quickly validate your REST APIs
• It delivers zero defect web applications with less effort in API testing
• To validate your web application no skilled resources are required, and it can generate documentation for your API specifications
• API mocks can be created in vREST with the help of Mock Server Functionality. User can directly start developing frontend using mock HTTP requests

Link to Vrest

8) Postman

Postman is a plugin in Google Chrome, and it can be used for testing API services. It is a powerful HTTP client to test web services. For manual or exploratory testing, Postman is a good choice for testing API.

• With Postman, almost all modern web API data can be extracted
• You can write Boolean tests within Postman Interface
• You can create a collection of REST calls and save each call as part of a collection for execution in future
• Unlike CURL, it is not a command line based tool, which makes this tool hassle free of pasting text into command line window
• For transmitting and receiving REST information, Postman is more reliable

Link to Postman

9) HttpMaster

It is a web development tool to automate web application testing, including API testing, service testing and website testing. It is primarily used as web API test tool to automate testing of web API calls.

HttpMaster is a best choice for API testing as it has got

• Several http methods (GET, POST, DELETE etc.)
• Dynamic parameters of various data types to make batches of different API requests
• Various validation types and advanced validation expressions

Download HttpMaster

10) Rest-assured

It’s a popular framework to test REST services in Java.

Learn more about Rest-Assured

Download HttpMaster

11) Karate DSL

Karate is a new API testing framework base on cucumber library. Karate DSL allows testers to write meaningful tests for web service using a domain-specific language.

Features:

• It supports configuration switching/staging, multi-threaded parallel execution
• Allows testing and generating reports just like any standard Java Project
• Possible to write tests even for the non-programmers
• The karate API testing tool allows re-use of payload-data and user-defined functions across tests.

Download link: https://github.com/intuit/karate

12) Rest Console

HTTP client and Request visualizer and constructor API testing tool. It helps developers to build, debug and test RESTful APIS.

Features:

• Construct POST or PUT body via raw input
• Easy query parameters creation
• Add custom headers through intuitive UI
• Keyboard navigation and shortcuts

Download link: https://github.com/ahmadnassri/restconsole

13) Hippie-Swagger

Hippie-swagger is a tool for testing APIS. It supports a clear assertion syntax, for extending test behavior. It also allows clean printing reports.

Features:

• It will fail test whenever swagger documentation is either erroneous or missing
• Readable and accurate assertation messages
• Validated parameters, request, response, pats, etc.

Download link: https://github.com/CacheControl/hippie-swagger

14) Pyresttest

PyRestTest is a Python-based REST API testing platform. It supports tests in JSON or YAML config files. Therefore, there is no code needed.

Features:

• It’s minimal dependencies, which allows easy deployment on-server for smoke tests/health checks
• This API testing tool will return exit codes on failure
• Pyresttest allows to generate/extract and validate mechanisms to build test scenarios

Download link: https://github.com/svanoort/pyresttest

15) Airborne

Airborne is an API automation testing tool used for testing Rest APIs.

Features:

• Airborne is a programming framework, so it has no user interface apart from the text file to create code
• To use airborne, you just need to remember a few key methods in the toolset and some ruby and rspec fundamentals

Download link: https://github.com/brooklynDev/airborne

16) JMeter

JMeter is used for functional API testing which includes needed to test an API. It also has features which help to boost the performance of API testing.

Features:

• It can be used for both static as well as dynamic resources performance testing
• It supports replaying of test results
• It can automatically work with CSV files. This helps test team to produce unique parameter values for API tests

Download link: http://jmeter.apache.org/

17) APIpray Inspector

Apiary allows monitoring the API during the design phase by capturing both request and response. It allows the user to write API blueprints and lets the user view them Apiary editor or Apiary.jo.

Features:

• Role-based access control over API documents.
• It allows to add and remove team members from API design projects
• API Blueprint management dashboard

Download link: https://apiary.io/

18) SOAP Sonar

SOAPSonar is an API testing and diagnostics platform for SOAP, XML, REST-based web services. Use external sources like Excel, MS SQL, Oracle or any ODBC Database for automated tests.

Features:

• Identifies Web Services vulnerabilities like Malware Threat and SQL Injection
• Web Service Functional Testing with Success Rule Framework and Concurrent Client Load Testing
• Native HP QC Integration, and supports integration with Hudson, Ant, and JUnit

Download link: http://www.crosschecknet.com/

19) API Science

API science allows monitoring the health, availability, and performance of web APIs. The tool enables monitoring Private, Partner and Public APIs. This tool allows the user to know if any API ever goes down so that needed action should be taken to bring it back up.

Features:

• It is multi-step & Powered by JavaScript
• Powerful reporting mechanism helps to get insights into historical trends and spot future issues.
• Supports JSON, REST, XML, and Oauth
• Helps to manage API supply chain

Download link: https://www.apiscience.com/

20) Apigee

Apigee is a cross-cloud API testing tool. It allows the user to measure and test API performance, supports and build API using other editors like Swagger. It offers security and governance policies across all APIs.

Features:

• Allows to design monitor, deploy, and scale APIs
• Easily create API proxies from the Open API Specification and deploy them in the cloud
• Identify performance issues by tracking API traffic, error rates, and response times

Download link: http://apigee.com/

21) Tricentis

Tricentis is a robust web services testing tool. Prominent API Testing features of Tricentis Tosca are –

• It supports a wide array of protocols including HTTP(s) JMS, AMQP, Rabbit MQ, TIBCO EMS, SOAP, REST, IBM MQ,NET TCP
• It integrates into the Agile and DevOps Cycle
• It uses model-based test automation that makes script maintenance easy.
• Enables end-to-end testing as API tests can be used across mobile, cross-browser, packaged apps, etc.

Tricentis’ 400+ customers include global names from the Top 500 brands such as ExxonMobil, HBO, Whole Foods, Toyota, Allianz, BMW, Starbucks, Deutsche Bank, Lexmark, Orange, A&E, Vantiv, Vodafone, Telstra and UBS.

Download link: https://www.tricentis.com/automated-software-testing-tool-trial/

22) SOAP UI

The most widely popular tool for API testing in the world, SoapUI allows you to test REST and SOAP APIs with ease – as it has been built specifically for API testing.

• Quick and Easy Test Creation: Point-and-click, drag-and-drop, functionality makes complicated tasks (like working with JSON and XML) simple
• Powerful data-driven testing: Load data from Excel, files, and databases to simulate the way consumers interact with your APIs
• Reusability of Scripts: Reuse your functional test cases as load tests and security scans in just a few clicks
• Seamless Integrations: Integrates with 13 API management platforms, supports REST, SOAP, JMS, and IoT

SoapUI Pro is used by thousands of leading companies around the world, including: Apple, Microsoft, Cisco, Oracle, HP, NASA, eBay, MasterCard, Intel, FedEx, and Pfizer

Download link: https://smartbear.com/product/ready-api/soapui/overview/

Behave - BDD test framework using Python

https://behave.readthedocs.io/en/latest/

https://github.com/behave/behave

Behavior-driven development (or BDD) is an agile software development technique that encourages collaboration between developers, QA and non-technical or business participants in a software project.

behave uses tests written in a natural language style, backed up by Python code.

First, install *behave*.

Now make a directory called "features/". In that directory create a file called "example.feature" containing:

# -- FILE: features/example.feature
Feature: Showing off behave

  Scenario: Run a simple test
    Given we have behave installed
     When we implement 5 tests
     Then behave will test them for us!

Make a new directory called "features/steps/". In that directory create a file called "example_steps.py" containing:

# -- FILE: features/steps/example_steps.py
from behave import given, when, then, step

@given('we have behave installed')
def step_impl(context):
    pass

@when('we implement {number:d} tests')
def step_impl(context, number):  # -- NOTE: number is converted into integer
    assert number > 1 or number == 0
    context.tests_count = number

@then('behave will test them for us!')
def step_impl(context):
    assert context.failed is False
    assert context.tests_count >= 0

Run behave:

$ behave
Feature: Showing off behave # features/example.feature:2

  Scenario: Run a simple test          # features/example.feature:4
    Given we have behave installed     # features/steps/example_steps.py:4
    When we implement 5 tests          # features/steps/example_steps.py:8
    Then behave will test them for us! # features/steps/example_steps.py:13

1 feature passed, 0 failed, 0 skipped
1 scenario passed, 0 failed, 0 skipped
3 steps passed, 0 failed, 0 skipped, 0 undefined

Now, continue reading to learn how to get the most out of behave. To get started, we recommend the tutorial and then the feature testing language and api references.

• behave documentation: latest edition, stable edition, PDF
• behave.example: Behave Examples and Tutorials (docs, executable examples).

JBehave  - Java BDD framework - eos

https://jbehave.org/

Create a BDD story with example scenarios 

A story is a collection of scenarios, each detailing different examples of the behaviour of a given increment of functionality of the system.

trader_is_alerted_of_status.story.

Map scenarios to Java methods with annotations

https://jbehave.org/reference/stable/developing-stories.html#writing

The scenario writer can specify the annotated methods in a POJO, i.e. without needing to extend/implement any JBehave class/interface. Of course, the Steps instances may inherit or implement other class/interface as required by the model of the application under test.

Configure Stories with data scenarios 

At the heart of the JBehave running of stories lies the Embedder, which provides an entry point to all of JBehave's functionality that is embeddable into other launchers, such as IDEs or CLIs. JBehave complements the Embedder with an Embeddable which represents a runnable facade to the Embedder.

JBehave allows many different ways to configure Embeddable Java classes that allow the parsing and running of textual stories.

JBehave provides two main Embeddable implementations:

• ConfigurableEmbedder: allows the specification of the Configuration and CandidateSteps.
• InjectableEmbedder: allows the injection of a fully specified Embedder.

JUnit-enabled Embeddables

JUnit is supported out-of-the-box via several Embeddables implementations:

• JUnitStory: provides a one-to-one mapping with the textual story via the StoryPathResolver.
• JUnitStories: provides a many-to-one mapping with the textual story paths explicitly specified by overriding the storyPaths() method.

In the case of one-to-one mapping, our abstract base TraderStory would look like:

public abstract class TraderStory extends JUnitStory {
 
    private final CrossReference xref = new CrossReference();
 
    public TraderStory() {
        configuredEmbedder().embedderControls().doGenerateViewAfterStories(true).doIgnoreFailureInStories(true)
                .doIgnoreFailureInView(true).useThreads(2).useStoryTimeoutInSecs(60);
        // Uncomment to set meta filter, which can also be set via Ant or Maven
        // configuredEmbedder().useMetaFilters(Arrays.asList("+theme parametrisation"));
    }
 
    @Override
    public Configuration configuration() {
        Class<? extends Embeddable> embeddableClass = this.getClass();
        Properties viewResources = new Properties();
        viewResources.put("decorateNonHtml", "true");
        // Start from default ParameterConverters instance
        ParameterConverters parameterConverters = new ParameterConverters();
        // factory to allow parameter conversion and loading from external
        // resources (used by StoryParser too)
        ExamplesTableFactory examplesTableFactory = new ExamplesTableFactory(new LocalizedKeywords(),
                new LoadFromClasspath(embeddableClass), parameterConverters);
        // add custom converters
        parameterConverters.addConverters(new DateConverter(new SimpleDateFormat("yyyy-MM-dd")),
                new ExamplesTableConverter(examplesTableFactory));
 
        return new MostUsefulConfiguration()
                .useStoryControls(new StoryControls().doDryRun(false).doSkipScenariosAfterFailure(false))
                .useStoryLoader(new LoadFromClasspath(embeddableClass))
                .useStoryParser(new RegexStoryParser(examplesTableFactory))
                .useStoryPathResolver(new UnderscoredCamelCaseResolver())
                .useStoryReporterBuilder(
                        new StoryReporterBuilder()
                                .withCodeLocation(CodeLocations.codeLocationFromClass(embeddableClass))
                                .withDefaultFormats().withPathResolver(new ResolveToPackagedName())
                                .withViewResources(viewResources).withFormats(CONSOLE, TXT, HTML, XML)
                                .withFailureTrace(true).withFailureTraceCompression(true).withCrossReference(xref))
                .useParameterConverters(parameterConverters)
                // use '%' instead of '$' to identify parameters
                .useStepPatternParser(new RegexPrefixCapturingPatternParser("%")) 
                .useStepMonitor(xref.getStepMonitor());
    }
 
    @Override
    public InjectableStepsFactory stepsFactory() {
        return new InstanceStepsFactory(configuration(), new TraderSteps(new TradingService()), new AndSteps(), new MetaParametrisationSteps(),
                new CalendarSteps(), new PriorityMatchingSteps(), new PendingSteps(), new ParametrisedSteps(), new SandpitSteps(),
                new SearchSteps(), new BeforeAfterSteps(), new CompositeSteps(), new CompositeNestedSteps(), new NamedParametersSteps());
    }
 
}

We then extend this base class with a class for each story, e.g. TraderIsAletedOfStatus.java, which maps to out textual story in same package.

In the case of many-to-one mapping:

public class TraderStories extends JUnitStories {
 
    private final CrossReference xref = new CrossReference();
 
    public TraderStories() {
        configuredEmbedder().embedderControls().doGenerateViewAfterStories(true).doIgnoreFailureInStories(false)
                .doIgnoreFailureInView(true).doVerboseFailures(true).useThreads(2).useStoryTimeoutInSecs(60);
        //configuredEmbedder().useEmbedderControls(new PropertyBasedEmbedderControls());
    }
 
    @Override
    public Configuration configuration() {
        Class<? extends Embeddable> embeddableClass = this.getClass();
        Properties viewResources = new Properties();
        viewResources.put("decorateNonHtml", "true");
        viewResources.put("reports", "ftl/jbehave-reports-with-totals.ftl");
        // Start from default ParameterConverters instance
        ParameterConverters parameterConverters = new ParameterConverters();
        // factory to allow parameter conversion and loading from external resources (used by StoryParser too)
        ExamplesTableFactory examplesTableFactory = new ExamplesTableFactory(new LocalizedKeywords(), new LoadFromClasspath(embeddableClass), parameterConverters);
        // add custom converters
        parameterConverters.addConverters(new DateConverter(new SimpleDateFormat("yyyy-MM-dd")),
                new ExamplesTableConverter(examplesTableFactory));
        return new MostUsefulConfiguration()
            .useStoryLoader(new LoadFromClasspath(embeddableClass))
            .useStoryParser(new RegexStoryParser(examplesTableFactory)) 
            .useStoryReporterBuilder(new StoryReporterBuilder()
                .withCodeLocation(CodeLocations.codeLocationFromClass(embeddableClass))
                .withDefaultFormats()
                .withViewResources(viewResources)
                .withFormats(CONSOLE, TXT, HTML_TEMPLATE, XML_TEMPLATE)
                .withFailureTrace(true)
                .withFailureTraceCompression(true)                
                .withCrossReference(xref)) 
            .useParameterConverters(parameterConverters)                     
            // use '%' instead of '$' to identify parameters
            .useStepPatternParser(new RegexPrefixCapturingPatternParser(
                            "%")) 
            .useStepMonitor(xref.getStepMonitor());                               
    }
 
    @Override
    public InjectableStepsFactory stepsFactory() {
        return new InstanceStepsFactory(configuration(), new TraderSteps(new TradingService()), new AndSteps(), new MetaParametrisationSteps(),
                new CalendarSteps(), new PriorityMatchingSteps(), new PendingSteps(), new SandpitSteps(),
                new SearchSteps(), new BeforeAfterSteps(), new CompositeSteps(), new NamedParametersSteps());
    }
     
    @Override
    protected List<String> storyPaths() {
        // Specify story paths as URLs
        String codeLocation = codeLocationFromClass(this.getClass()).getFile();
        return new StoryFinder().findPaths(codeLocation, asList("**/trader_is_alerted_of_status.story",
                "**/traders_can_be_subset.story"), asList(""), "file:" + codeLocation);
    }
         
}

JUnit AnnotatedEmbedderRunner

JBehave also provides an implementation of JUnit's Runner, AnnotatedEmbedderRunner, which is runnable via JUnit's @RunWith annotation:

@RunWith(AnnotatedEmbedderRunner.class)
@Configure(storyLoader = MyStoryLoader.class, storyReporterBuilder = MyReportBuilder.class, 
        parameterConverters = { MyDateConverter.class })
@UsingEmbedder(embedder = Embedder.class, generateViewAfterStories = true, ignoreFailureInStories = true, ignoreFailureInView = true)
@UsingSteps(instances = { TraderSteps.class, BeforeAfterSteps.class, AndSteps.class, CalendarSteps.class,
        PriorityMatchingSteps.class, SandpitSteps.class })
public class TraderAnnotatedEmbedder implements Embeddable {
 
    private Embedder embedder;
 
    public void useEmbedder(Embedder embedder) {
        this.embedder = embedder;
    }
 
    @Test
    public void run() {
        embedder.runStoriesAsPaths(new StoryFinder().findPaths(codeLocationFromClass(this.getClass()).getFile(),
                asList("**/*.story"), asList("")));
    }

Integration with other frameworks

As remarked above, JBehave does not impose any tie-in with any framework to run stories. It only requires access to the Embedder to run the stories. The following snippet shows, for example, how to use SpringJUnit4ClassRunner to compose and inject steps instances and them the stories:

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations = { "/org/jbehave/examples/trader/spring/steps.xml" })
public class AnnotatedEmbedderWithSpringJUnit4ClassRunner {
 
    @Autowired
    private TraderSteps traderSteps;
 
    @Autowired
    private BeforeAfterSteps beforeAndAfterSteps;
 
    @Test
    public void runStoriesAsPaths() {
        embedder().runStoriesAsPaths(storyPaths());
    }
 
    @Test
    public void findMatchingCandidateSteps() {
        embedder().reportMatchingStepdocs("When traders are subset to \".*y\" by name");
        embedder().reportMatchingStepdocs("Given a step that is not matched");
    }
 
    private Embedder embedder() {
        Embedder embedder = new ClasspathTraderEmbedder();
        embedder.useStepsFactory(new InstanceStepsFactory(embedder.configuration(), traderSteps, beforeAndAfterSteps));
        return embedder;
    }
 
    protected List<String> storyPaths() {
        StoryFinder finder = new StoryFinder();
        return finder.findPaths(codeLocationFromClass(this.getClass()).getFile(), asList("**/*.story"), asList(""));
    }
 
}

Run Stories 

JBehave provides fully annotatation-based support for specifying configuration and dependency injection. The running stories will go into more details of the different ways to run stories. Or if you want to learn more about JBehave's step matching mechanism, you'll want to explore the concept of candidate steps in more detail.

https://jbehave.org/reference/stable/running-stories.html

Running Stories

JBehave is designed to be embedded in different development environments. The JBehave Core module contains support for running stories as JUnit tests - which can be run either in your favourite IDE or in your command-line build that supports JUnit tests. Other unit testing frameworks, e.g. TestNG or Spring Test, can also be used very easily, c.f. FAQ.

You can also run stories using the JBehave Ant tasks or Maven goals, which provide a facade around the Embedder functionality.

JBehave provides the Embedder as the core entry point to running stories. The Embedder can be configured in multiple way, c.f. the configuration.

Multiple running modes
Stories can be run in different modes:

• as embeddables: An Embeddable is an abstraction of a running mode, which can run a single story, e.g. if extending JUnitStory or can run multiple stories, e.g. if extending JUnitStories. The Embeddable instances allow their configuration.
• as paths: once we have specified an instance of an Embedder, with its configuration, we run stories specified by paths.
  with annotated embedder runner: an AnnotatedEmbedderRunner is an implementation of the JUnit Runner, which allows annotated specification of the Embedder to be used to run the stories. C.f. configuration on

how to use annotatations to configure the Embedder.
with annotated path runner: an AnnotatedPathRunner is an extension of AnnotatedEmbedderRunner which creates a JUnit suite containing each story path and resolving the path to a human readable form that plays nicely with any IDE that provides JUnit integration.
Running Stories in a framework-neutral way

The Embedder can also be used to run the stories in any IDE environment, without the need to extend any base class. In the following example we use JUnit annotatation to denote different running instances:

public class TraderStoryRunner {
 
    @Test
    public void runClasspathLoadedStoriesAsJUnit() {
        // Embedder defines the configuration and candidate steps
        Embedder embedder = new TraderEmbedder();
        List<String> storyPaths = ... // use StoryFinder to look up paths
        embedder.runStoriesAsPaths(storyPaths);
    }
 
    @Test
    public void runURLLoadedStoriesAsJUnit() {
        // Embedder defines the configuration and candidate steps
        Embedder embedder = new URLTraderEmbedder();
        List<String> storyPaths = ... // use StoryFinder to look up paths
        embedder.runStoriesAsPaths(storyPaths);
    }
 
}

where the TraderEmbedder/URLTraderEmbedder define the configuration using the loading from the classpath and URL resources, respectively. E.g.:

public class TraderEmbedder extends Embedder {
 
    @Override
    public EmbedderControls embedderControls() {
        return new EmbedderControls().doIgnoreFailureInStories(true).doIgnoreFailureInView(true);
    }
 
    @Override
    public Configuration configuration() {
        Class<? extends TraderEmbedder> embedderClass = this.getClass();
        return new MostUsefulConfiguration()
            .useStoryLoader(new LoadFromClasspath(embedderClass.getClassLoader()))
            .useStoryReporterBuilder(new StoryReporterBuilder()
                .withCodeLocation(CodeLocations.codeLocationFromClass(embedderClass))
                .withDefaultFormats()
                .withFormats(CONSOLE, TXT, HTML, XML)
                .withCrossReference(new CrossReference()))
            .useParameterConverters(new ParameterConverters()
                    .addConverters(new DateConverter(new SimpleDateFormat("yyyy-MM-dd")))) // use custom date pattern
            .useStepPatternParser(new RegexPrefixCapturingPatternParser(
                            "%")) // use '%' instead of '$' to identify parameters
            .useStepMonitor(new SilentStepMonitor());                               
    }
 
    @Override
    public InjectableStepsFactory stepsFactory() {
        return new InstanceStepsFactory(configuration(), new TraderSteps(new TradingService()), new BeforeAfterSteps());
    }
 
}

Running Remote Stories

JBehave supports running both local and remote stories. To run remote stories, we need to use a URL-based loader with an appropriate remote code location. The difference w.r.t. the local run is minimal:

public class RemoteTraderStories extends TraderStories {
 
    @Override
    public Configuration configuration() {
        return super.configuration()
               .useStoryLoader(new LoadFromURL())
               .useStoryReporterBuilder(
                       new StoryReporterBuilder()
                           .withCodeLocation(codeLocationFromURL("http://jbehave.org/reference/examples/stories/"))
                           .withDefaultFormats()
                           .withFormats(CONSOLE, TXT, HTML, XML));
    }
 
    @Override
    protected List<String> storyPaths() {
        // Specify story paths as remote URLs
        String codeLocation = codeLocationFromURL("http://jbehave.org/reference/examples/stories/")
                .toExternalForm();
        return asList(codeLocation + "and_step.story");
    }
 
}

Ignoring Failures Running Stories

By default, the story runners are configured to fail-fast, i.e. the execution will stop at first failed story (but will complete execution of the all the scenarios in the story first). To allow the generation of a complete stories view (reporting how many stories failed), the runners need to be enabled to run stories with ignoreFailureInStories flag set to true. In this way, all stories will run and the failure will be assessed only during the view generation. If any stories failed, the build will fail correspondingly. Should the need to ignore failure in the view arise (although generally not recommended), one can set the ignoreFailureInView flag to true.

View Reports 

https://jbehave.org/reference/stable/reporting-stories.html

Reporting is an essential element of BDD as it allows to monitor the outcome of the stories that have been run. At the heart of JBehave's reporting is the StoryReporter, to which events are reported as they occur.

Currently, the story reporters supported are:

• ConsoleOutput: a text-based console output
• IdeOnlyConsoleOutput: a text-based console output, which only reports events when running in IDEs (Eclipse and IDEA supported).
• TxtOutput: a text-based file output
• HtmlOutput: an HTML file output
• HtmlTemplateOutput: an HTML template-based file output. By default, a Freemarker-based template processor is used but a different implementation based on a different templating system can be provided.
• XmlOutput: an XML file output
• PostStoryStatisticsCollector: collects statistics and stores them as properties after story is run
• DelegatingStoryReporter: used by the StoryReporterBuilder to delegate to any number of reporters as a proxy.

The StoryReporterBuilder allows to configure multiple story reporters with pre-configured formats: CONSOLE, TXT, HTML, HTML_TEMPLATE and XML.

JBehave tips 

create JBehave REST api templates in Groovy dynamically to eliminate custom mappings for each BDD test - only need self documenting BDD scripts that are dynamically loaded to Groovy smart BDD test classes

Step-by-step guide for Example

sample code block

Recommended Next Steps

Related articles