I mentioned that I would have a follow-up rant about Robot Framework when previously venting about micromanaging.

1/

So, what about it?

It's an automated acceptance testing tool. The idea is to represent test cases in a human friendly format that can have associated code filled in behind it to automate the verification process.

So far, so normal.

2/

In essence it's like Cucumber except it uses a tabular format for writing tests. This makes tests easy to parse for code but makes them hard to read and awkward to write.

3/

As I've used specification languages like Cucumber more I've come to the conclusion that they're a waste of time over code.

The biggest problem is that by trying to be both a "human" language and code neither Robot nor Cucumber work well as either.

4/

As a human language the text is stilted and awkward to read. As a computer language it's equally awkward due to a lack of distinction between actions, checks and verification.

For example:
GIVEN I am logged in as admin
Vs
THEN I am logged in as admin

4/

THEN looks clear but might not be. It could be verifying that the user is a specific account, the user is a specific role, etc. Human language is context sensitive but that context cannot be carried into code in a natural manner leading to weirdly redundant rephrasing

5/

I am ??? logged in
I am logged in as ???
I am logged in as an ???
Etc.

There are ways around this but you're required to think of your tests like code and not like documentation of businesses requirements.

You cannot use natural language. The main premise of BDD is broken

6/

Furthermore is the issue with GIVEN. Despite natural language indicating that GIVEN describes actions that need to be taken before a test it means that "I am logged in as" means *actually logging in* and not merely checking something related to logging in.

7/

Thus more contrived rephrasing to ensure the code works even though it should be unnecessary when writing a business requirement.

The alternative is to force human context into your code by burying a state machine in your procedures but that gets real painful real quick

8/

IMO the proper solution is to write test plans in terms of functional requirements and describe what is supposed to happen for each requirement with *real* natural language.

Use real code to automate a d tie your automation to the requirements using software

9/

You could name your automated cases identically to your business requirements. Jasmine is good for this:
it('should deliver bacon when button is pressed', ...);

Or use metadata:
[TestCaseId("Auth.1.1.5")]
void VerifyADRoleMapping()

Or put the requirement docs in code

10/

Anyway, back to Robot.

The general issues with BDD-like interfaces means a lot of code needs to be written under the surface to make the tests look somewhat sane. Despite this no business people ever read the tests.

BAs work with English and math, not QA-nerdlish.

11/

Coding for Robot is OKish. Each row in the table is simply a Python function call. The first cell is the function name, every other cell is an argument.

The pain comes in when we need to work with the framework itself.

12/

Someone somewhere decided that the best choice for the framework was to make literally everything with the visitor pattern then abstract everything to hell and back on top of it.

A test framework, especially one that works the way Robot does, is not complicated.

13/

Fundamentally a test plan is a shallow, but wide, tree. I suspect that this is where the visitor pattern came to be used. The original Gang of Four example walked a tree using the pattern even though the tree was not actually the point of the pattern in the first place!

14/

The point of a visitor patten is to decouple an iterator from a set of different types of objects. A level editor could use this with one visitor for serialization and another for rendering. (Note: this is a really bad idea but I digress)

15/

The Visitor pattern is handy when you iterate over a set of objects for many different purposes and need to add new types over time.

A test framework is not such an animal.

16/

Your types are suites, tests and steps. Suites also optionally have fixtures for themselves and individual tests with steps. Maybe some metadata. No additional types needed.

Actual actions are similarly limited to searching, execution and maybe serialization.

19/

So Robot's guts are filled to the brim with unnecessary complexity. It's difficult to use, difficult to have a deep understanding of its workings, and, because of that, difficult to extend.

In all, a thoroughly unpleasant experience in using and supporting it.

20/20

I just realized I have two 4s in this thread. Oops