Buch “Test Automation Fundamentals”

Einleitung

In diesem Beitrag geht es um das Buch “Test Automation Fundamentals” von Baumgartner et al., welches im dpunkt.verlag erschienen ist und für 39,90 Euro gelistet wird. Ich selbst hatte durchaus großes Interesse an dem Buch, da ich mich im Rahmen meiner Projekte PIRA und MetaCG in den letzten ca. 1,5 Jahren vermehrt mit dem Thema Software-Testing befasst habe. Jedoch bin ich dabei eher einer Learning-by-Doing-Strategie gefolgt und habe mich auch an den Testregimen anderer Open-Source-Projekte bedient. Gerade daher war ich gespannt, wie ein (Lehr-)Buch die Herangehensweisen und Inhalte vermittelt.

Das Buch richtet sich an Test Automatisierungs Ingenieure:innen, Software Tester:innen, Software Entwickler:innen, sowie an Lehrer:innen und Schüler:innen. Es ist in englischer Sprache verfasst und ich werde in dieser Rezension darauf verzichten, etwaige Zitierungen ins Deutsche zu übersetzen.

Meine Erwartungen an das Buch waren durchaus hoch, in dem Sinne, dass ich vieles über das fundamentale Design von Software Tests lernen werde, und insbesondere auch strukturierte Herangehensweisen an das Testen vermittelt werden. Die Rückseite des Buches beschreibt den Inhalt, dass das Buch “provides a complete overview of how to design test automation processes and integrate them into your organization or existing projects. It details functional and technical strategies and goes into detail on the relevant concepts and best practices. The book’s main focus is on functional system testing.” [Baumgartner et al.]

Inhalt und Eindruck

Das Buch startet mit einer längeren Erklärung, warum automatisiertes Testen gut und mitunter notwendig ist. In diesem Kapitel werden auch Erwartungen gesetzt, bzw. die Grenzen aufgezeigt, welche Aspekte das Buch nicht behandelt. Insgesamt sehe ich das Kapitel als notwendig und sinnhaft an. Ich muss jedoch auch sagen, dass es für mich sehr lang erschien, da viele Aspekte, von denen ich bereits überzeugt bin, noch einmal in voller Länge erwähnt und begründet wurden. Hier ganz klar der Hinweis, dass ich mich mit der grundlegenden Sinnhaftigkeit, der Abwägung von Investition und möglichen Benefit, etc pp, schon hinreichend befasst hatte und vieles einfache Wiederholung war. Für eine Einführung in das Thema ist dieses Kapitel aber sowohl fundamental wichtig, als auch sinnhaft und letztlich überzeugend und verständlich geschrieben.

Danach gibt das Buch einen sehr guten Einblick in die technischen Überlegungen, wann und wie man Testautomatisierung in einem Projekt einbringen kann, und, wie man nach Möglichkeit das richtige Werkzeug wählt. Insbesondere werden eine ganze Reihe von technischen Überlegungen angestellt, die mir neu waren, und die ich für wichtig halte. Was mir sehr gut gefallen hat ist, dass die Autoren Beispiele und Beispielszenarien geben und die Technologiewahl mit ihren Gründen. Hier hat das Buch meine Erwartungen de facto übertroffen und ich bin sehr froh, es gelesen zu haben.

Hierauf folgend geht das Buch in großem technischen Detail auf die Realisation einer Testautomatisierung ein. Es werden viele verschiedene Projekt-Szenarien als Beispiele genannt und welche Hindernisse es geben kann, oder welche positiven Auswirkungen entstehen können. Dieser Teil ist, in meinen Augen, der Schwerpunkt – zumindest in technischer Hinsicht – und die Autoren geben einen verständlichen Einblick in die Technologien und ihre Vor- und Nachteile. Es sei hier angemerkt, dass alle Beispiele in dem Buch durchaus komplexe Softwaresysteme betrachten. Damit meine ich insbesondere Systeme, die aus diversen, in verschiedenen Sprachen implementierten, auf verschiedenen Hardwaresystemen ausgeführten Komponenten bestehen. Dieser Aspekt hat für mich eine weitere Dimension in meinen Überlegungen geöffnet, da meine eigenen Projekte recht kompakt sind und nur wenige Zielsysteme haben.

Weitere Aspekte, die im Verlauf des Buches noch behandelt werden, sind potentielle Stolpersteine für die Einführung eines Testautomatisierungssystems. Die Autoren geben hier sowohl technische Hindernisse, als auch organisatorische Hindernisse an. Als Teil dieser Sektion werden insbesondere auch die Risiken beleuchtet, die die Einführung einer komplexen Testautomatisierung mit sich bringen kann.

Ebenfalls wichtig, und im Buch gut beleuchtet, ist das Thema des Reportings einer Testinfrastruktur und was mit dem Wissen von Defekten geschehen soll. Dies ist vermutlich eine der spannendsten Sektionen außerhalb der technischen Teile, da es auch um Kultur und Prozesse geht. Letztlich nützt die beste Testinfrastruktur nichts, wenn Testergebnisse nicht ernst genommen werden.

Abschließend werden die Aspekte der Testvalidierung und die kontinuierliche Verbesserung des Testsystems behandelt. Insbesondere der Aspekt des kontinuierlichen Verbesserungs gefällt mir sehr gut. Hier sehe ich auch einen wichtigen Beitrag, denn ich bin mittlerweile der Meinung, dass es besser ist, man fängt an und verbessert kontinuierlich, verglichen mit man bewegt sich gar nicht, da man keine perfekte Lösung parat hat.

Fazit

Zusammenfassend kann ich nur sagen, dass mir das Buch sehr gefallen hat und ich viel gelernt habe. Insbesondere der Blick auf potentiell historisch gewachsene, große Projekte, bei denen die Einführung einer Testautomatisierung selbst ein signifikantes Projekt ist, hat mir Freude gemacht. Viele dort relevante Aspekte kamen in meinem bisherigen Software-Entwickler-Dasein noch nicht zum Tragen. Wer sich also nicht scheut, ein in Englisch geschriebenes Buch zu lesen und sich für das Thema Testautomatisierung interessiert, dem sei das Buch durchaus ans Herz gelegt.

Mir wurde für diese Rezension ein Rezensionsexemplar zur Verfügung gestellt.

MetaCG — Branching Model

After some history of the MetaCG project, let’s look a little more into technical things. First up is our branching model: What we use, and why we use it. As a general note, we use a private Gitlab instance for our development work, but have recently thought again about moving everything to GitHub. We have not finally decided what would be better for the project and how much work is needed to port our current CI to GitHub actions.

But, let’s get into the branching model for now.

Branching Model: Gitflow

We use the Gitflow branching model for the development of MetaCG. At least basically, and with a few “modifications”, compared to the original post linked here. There are some general considerations for our project: (1) It must be possible to develop multiple ideas and features for papers while working on student theses simultaneously, (2) we want to keep development private (general paranoia in academia for steeling ideas, no matter how relevant this actually is), (3) we want to maintain a public mirror so interested people can get access to our project, (4) the project should be traceable in its development to link software used for papers, and (5) connecting implementation with issues in our issue tracker to know of bugs and resolve limitations.

Given these considerations, it seemed to me that Gitflow is a good starting point. In Gitflow one maintains multiple branches for features, the development work, and the current version of the project. In our case, we maintain two main branches: devel and master and however-many feature branches we need. Our master branch is automatically mirrored from the private development instance to our public repository at github.com/tudasc/MetaCG. You can think of it as a release branch.

The other main branch is the devel branch. It is the branch to move the project forward and integrate features into. These features are developed on feature branches, that I’ll talk about a little in the next paragraphs. Let me just say that we do not develop hotfixes as part of our development model. We fix issues in devel and, if ‘critical’, push to master, i.e., release a new version. However, since our software is mainly focused on research, we do not see that any of our fixes are actually ‘critical’.

Finally, let’s talk about feature branches. Our feature branches are used for two things: feature development and theses projects. This differentiation may be the biggest difference in the intention of the use for feature branches in the Gitflow model. Yet, it is surely motivated from the academic setting that MetaCG is developed in.

Feature development: This is what one would expect from a feature branch. The feature is implemented while paying attention to software quality, usability and similar things. This means that the goal of a feature is to make the software more usable and more “a product”. This is different from theses projects.

Theses Projects: At universities, one of the important things is to allow students to work on projects for their Bachelor and Master degree. In my case, these projects are typically tied around MetaCG and/or PIRA. The thesis goal is then briefly explained in our issue tracker and a feature branch for this issue is created. Theses projects are typically not meant to be user friendly, but to demonstrate a particular idea and approach. One example could be the development of PIRA’s automatic load-imbalance detection for MPI applications. Once the theses is submitted and presented, these projects are, subsequently, treated like features, and shall be cleaned up and improved for usability before being re-integrated into the project. This is, however, no longer part of the thesis, but is usually done as a student research assistant.

All feature branches follow the same naming scheme: feat/<ISSUE-ID> so that it is easy for a project contributor to know what the newly introduced code is meant to do and address and the final merge request can clearly communicate what the code does.

Finally, and I’ll talk about that more in a later post, the preparation of new releases is also done on a feature branch / multiple feature branches. These feature branches, however, have a distinguishable name.

If you are curious about the decisions, think that there are better ways to do it, or would like to contribute, reach out to me! Potentially shoot me a message or tag me on twitter.

MetaCG Development Details Mini Series

This is the first post of a mini series that I thought may be interesting to do. It will go a little into the details of our development model for the MetaCG library and their reasons. The things we do here are obviously by no means “the” right thing to do for every project, but maybe the perspective on the different topics is relevant for you and your decision making for your own project. The series will consist of three parts.

  • Branching Model: In the first part we provide an overview of our branching model and why we chose it.
  • Development and Testing: The second part will go over our development work. This is probably the largest part of the mini series and may be split up into multiple parts itself.
  • Release Management: In the third part we provide an overview of our release management, i.e., which releases we do, how they are prepared and how they are finalized.

However, let’s first have some history on MetaCG as this will be the reason for some of the decisions that were and are taken.

MetaCG — History

MetaCG started (probably late 2013) as a nameless mock-tool for smart instrumentation tools and heuristics within our work on the InstRO tool. It was used to construct the call graph for a program given an unfiltered Score-P profile. An unfiltered Score-P profile is a runtime profile recorded with the Score-P tool that contains all call edges from the original source code executed at runtime. Given this call graph, the tool would then evaluate the heuristics that were published in the paper “Calltree-Controlled Instrumentation for Low-Overhead Survey Measurements”.

Then, during my PhD time, I evolved the code base of the nameless mock-tool starting in 2017 into what was then referred to as MetaCG. The code base was, however, still almost exclusively focused on the use within the PIRA tool and our work on iterative performance instrumentation refinement that we published in our paper on PIRA.

After our paper on “Automatic Instrumentation Refinement for Empirical Performance Modeling”, we decided to evolve MetaCG into its own tool that can be used more easily outside of the PIRA context. The idea behind this was to ease our research tools development should we want or need to pass information from the source level to the LLVM level. We wrote a paper about MetaCG and used the now-available whole-program call-graph information for an extended analysis in our paper “Towards compiler-aided correctness checking of adjoint MPI applications”.

Since then we have worked on the MetaCG code base to separate it more and more from the initial use case within PIRA. This means that it is now evolving into a set of libraries and tools that build on top of these libraries. One tool is the (also evolving) analyzer used within PIRA and a more recently created tool is the analyzer built as part of the CaPI project.

This evolution of the MetaCG code base has lead to several significant changes and re-designs. Many of these changes were really motivated and necessary while we evolve from a special-purpose mock-up tool, to a prototype tool for one particular context, to a set of libraries for tool construction. What I learned in the process may also make it into an article on this website eventually.

As one important side note, coming from academia, the possibility for (a) working on paper ideas and (b) students to work on parts of MetaCG for a thesis was and is very important. This can be seen in the branching model and other areas of the tool and library throughout the mini-series.

Buch: “Projektmanagement kurz & gut”

Zusammenfassung

Das Buch mit dem Titel “Projektmanagement kurz und gut” aus O’REILLYs Taschenbibliothek von Daniel Brönimann und Christoph Bommer richtet sich in meinen Augen an Personen, die bereits Erfahrung im Projektmanagement gesammelt haben. Es fokussiert den Blick und das vorhandene Wissen auf die Dinge auf die es im Projektmanagement tatsächlich ankommt: Das Projekt! Hierzu orientiert sich die Gliederung des Buches an den Projektphasen. Von der Vorbereitung und dem Start des Projekts, über die Realisierung bis zum Abschluss des Projekts. Darüber hinaus betrachtet es kurz die Rollen und Beziehungen im Projekt. Es schliesst mit einer kurzen Einordnung von Agilität in Projekten sowie einer kurzen Starthilfe. 

Eindruck

Das Buch ist mit seinen 132 Seiten (inklusive Literaturverzeichnis und weitergehenden Hinweisen) relativ kurz und lässt sich insgesamt sehr gut lesen, da die Sprache letztendlich einfach gehalten ist. An der ein oder anderen Stelle habe ich mich ertappt wie ich ein wenig über das ein oder andere deutsche Wort schmunzeln musste — vermutlich auch, weil ich normalerweise auf Englisch lese und schreibe.

Projektmanagement — kurz und gut.

Das Buch beleuchtet die verschiedenen Projektphasen von Vorplanung, über Start, Durchführung, sowie das Projektende. In jeder der Phasen stehen andere Schwerpunkte im Vordergrund, und das Projektmanagement muss verschiedene Aufgaben bewältigen. Für diverse Aufgaben, z.B., beim Thema Aufwandsschätzung oder Fertigstellungsgrad messen, geben die Autoren dem Lesenden beispielhaft Methoden an die Hand. Wenngleich die Methoden nicht en-detail erklärt werden, so kann man sowohl ihren Einsatz, als auch ihren Wert erkennen.

Auf den Punkt gebracht, erinnert das Buch den Lesenden immer wieder an: (1) Das Projekt steht im Vordergrund. (2) Das Projektmanagement muss in der Zukunft denken und in der Gegenwart handeln. (3) Eine der wichtigsten Ressourcen sind Menschen, deren Gefühle und Selbstwahrnehmungen man ernst nehmen muss.

Ich selbst fand die Fokussierung auf das wichtigste im Projekt gut, obwohl ich mir an dieser Stelle gewünscht hätte, dass ein bisschen mehr Methodik dargelegt wird. Damit meine ich Herangehensweisen und Techniken um die eine oder andere Situation bei der Planung zu lösen. Das Buch ist nicht frei von Methodik, es hat jedoch ganz klar einen anderen Fokus. Es geht in dem Buch um die Rückbesinnung auf das Wichtigste im Projektmanagement: das Projekt, und dass es die Aufgabe des Projektmanagements ist, dieses zu einem guten Ende zu führen.

Fazit

Was ist eigentlich wichtig, wenn ich im Projektmanagement agiere? Was wird von mir erwartet und worauf sollte ich meine Energie fokussieren? Wenngleich meine Erwartungen nicht vollkommen erfüllt wurden, bin ich der Meinung, dass das Buch wichtige Punkte anspricht, um sich nicht im Methoden Dschungel zu verirren.

Insbesondere für den, in meinen Augen, erschwinglichen Preis von 14,90 € kann ich das Buch denjenigen Empfehlen, die den Blick nochmal auf das richten möchten was im Projektmanagement zählt: Das Projekt und nicht nur die Prozesse drum herum.

Mir wurde für diesen Beitrag ein Rezensionsexemplat zur Verfügung gestellt.

Audio Setup Improvements

After almost a full year of working from home and numerous video conferences in different software solutions, I have finally upgraded my audio equipment. Initially I wanted to revive my old Tascam US122L, but it seems that all solutions I found to get that device working in Linux are outdated by now.

TL;DR
I bought a Focusrite Scarlett 2i2 3rd Gen USB audio interface*, a the t.bone condenser microphone, and a desk stand for the microphone. I am very satisfied with the overall setup quality and the sound it produces, although I would buy a different desk stand next time.

Longer Story

I have been looking into beginner’s home recording equipment for some time due to my general interest for it. Since, generally speaking, time is the most limiting factor, I never considered buying any of the equipment, as I thought I would never use it. With the pandemic changing the way I work and interact with my colleagues etc, these things have changed, and I finally had an excuse to spend some money.

I had two requirements that the audio interface needs to fulfill:
1. I want a two channel / stereo interface with somewhat decent pre-amps, and,
2. it needs to work on Linux (and potentially Windows).

After spending some time searching the internet, I finally came by the Scarlett series by Focusrite. It seems that their (smaller) interfaces are USB class compliant, meaning that they work with the Linux built-in USB stack – which I can confirm (Manjaro Linux, Kernel 5.11). The reviews for the pre-amps read decent, and it is not excessively expensive. So far, I can only agree and conclude that I like the sound, and knobs and casing look and feel very high quality.

As microphone, I went with more of a budget solution: the “the t.bone SC 400” condenser microphone. Though it is comparably affordable, I think it offers a decent and rich sound. I do feel, however, that it lacks some airiness in the high frequencies.

If you are interested in a little voice demonstration, the audio setup is the one I use for my twitch live streams. So don’t hesitate and join me for some demo of the sound on Wednesday night.

I’m now on twitch!

Yes that’s right! I’m now on twitch!

I have lately started to stream some programming and building games live on twitch.tv/jplehr and found that quiet relaxing and nice. So I want to continue it with a schedule. I’m happy if people join-in to chat, learn more about the software that I work on, maybe about programming in general, or whatever we want to chat about.

Wednesday from 09 PM CET: Programming

Every Wednesday from 09.00 pm CET to about midnight, I am continuing on one of the software packages that I introduce on this website. Currently, I am mostly working on MetaCG or PIRA. Both can also be found on Github.

Sunday from 09 PM CET: Open Stream

I’ll probably also stream on Sunday’s, from 09.00 pm CET to about midnight. But on Sundays, it’ll not only be programming. Other stuff I want and can stream, given the current computers I have, are OpenTTD or Cities Skylines. So some relaxing building and development games. That may change obviously. 😉

So, if you want to know what’s coming up, follow me on twitter or twitch and receive updates and notifications on what I’ll be doing.

MetaCG – Annotated Whole-Program Call-Graphs

MetaCG: A tool suite for whole-program inspection and automatic performance instrumentation generation. It brings a call-graph extractor using Clang Tooling, a whole-program call-graph library that is serializable to a json-based format and allows annotation with user-defined meta data, hence the name MetaCG. Finally, a call-graph validation tool that uses a Score-P profile to determine which edges are missing.

Repository: https://github.com/tudasc/metacg

In our work on PIRA, we realized that we need a whole-program call-graph representation that we can analyze and annotate with user-defined information. There are obviously multiple ways to do that, and we decided (more or less well-informed) to implement it as a library together with a toolchain to extract the call graph from C/C++ code using Clang Tooling. To evaluate its completeness we figured that it is easiest for us to use instrumentation-based profiling data using Score-P. This is a dependence of PIRA anyway, and at that time the call-graph library was only used within PIRA. Later on, we realized that we want to use whole-program reachability information in other tools as well, and think that the call-graph library of PIRA is a reasonable abstraction.

So we started MetaCG as a more general software package.

The software package is written in C++, uses the CMake build system and is licensed under a BSD 3-clause license. It comes with five software components.

MetaCG Library: The fundamental call-graph representation. A lightweight, bi-directional graph of which the function nodes can hold user-defined information in MetaContainers. The graph can be serialized into json, in which case the MetaContainers are output to every function node with a specified key such that they can be identified in the json file later on, e.g., by a subsequent analysis tool. Currently, it does not contain explicitly modeled edges, which limits its expressiveness to some extent. However, this is a feature that is planned and will be added when time permits.

CGCollector: The Clang-based call-graph extractor. It processes the abstract syntax tree and obtains information about the class hierarchy, call relations, and other source-level information that a user needs. The latter is done through the MetaCollector extension point, i.e., for every source information that should be annotated, a new MetaCollector is derived, obtains the desired information, and attaches it for a specific tool to the MetaCG.

CGMerge: CGCollector works on a single translation unit at a time, hence, the partial call graphs need to be merged. This is done, similar to linking for a binary, with CGMerge. It takes all translation-unit local files and merges them. It needs some strategy to resolve potential multiple entries in the meta data, i.e., data generated from a MetaCollector, fields, hence, the user is required to provide them.

CGValidate: The tool gets a MetaCG and a Score-P profile in Cube format (please note, it needs to be a full profile, i.e., with all functions marked as inline etc), and checks which edges are not present in the MetaCG. This allows a user to validate that all potential function calls are contained, and if not, CGValidate can patch the missing edges into the MetaCG.

PGIS: The PIRA analyzer that performs call-graph analysis to generate low-overhead performance instrumentation for subsequent measurement with Score-P.

If you are curious, please check it out, and report issues and bugs in the issue tracker on Github. I also plan to write more articles here that explain some components or use-cases in more detail.

The development currently takes place in a university-hosted Gitlab instance, hence, not every feature that is being worked on is already public. Should you be interested in our progress or even in contributing to the project, please also open an issue on Github and we can figure out how you get access.

Changes to this website

I have decided that I want to use this website not only for notes taking – which quite honestly never really happened – but write about what I do. While this includes some of the computer science stuff that is here already, it will open up the variety of things a little bit.

So: what to expect?
Well: I don’t know for sure, yet. But it will be closer to what you can find on my twitter.

Computer Science and High Performance Computing

This part, which was the only thing I considered for now, will still be here. Most of this is also relevant for my daytime job and my research.
Actually, I want to increase this to not only little notes that helped me from time to time but to what I do. Also, I figured it can be interesting to provide pointers to some other great work of people that I work or just some other tools that I find helpful and enjoy.

Hence, I will probably introduce a *software* section (or tag) that includes (research) software that I use, enjoy, develop, want to promote, etc.

Sports

I also enjoy doing sports quite a bit, and decided that I am going to share some of what I do on this website as well. This probably includes some links to training data as well as gear that I use. No false expectations here: I’m not nearly as much of an athlete as I would like to be. But I do some sports and want to share some of my experiences.

Other free-time activities

Finally, I’ll also add some other free-time activities. I have a couple of very, very easy woodwork things on my list that I may share little stories of. However, this will be (I guess) the smallest part of what I add to the website.

HPC Hallway

As a result of the Covid 19 / Coronavirus situation, an informal weekly meeting, known as the HPC huddle, established itself. It offers a room for discussion around HPC, AI, cloud, and other, related, topics.

To preserve the shared links, we decided to create a Slack. You can find the join link at www.hpc-hallway.org

During conferences, such as ISC or SC, the meeting frequency may increase, to offer more room for discussion and informed speculation.

Determine max. memory consumption of process

In a recent project, we needed to measure the increase in memory consumption for an application process. How to obtain “the right values” for this depends on the actual scenario and, apparently, is not straight forward in all cases.

Let me first describe the scenario a little more: We want to obtain measurements for both fully serial and MPI parallel applications. These applications are run in (1) an unchanged (vanilla), (2) an instrumented version (version 1) and (3) a version, which uses LD_PRELOAD to sneak-in another library that overloads MPI functions to do additional work (version 2).

More precisely what we want:

  • A way to obtain reliable measurements for the different configurations, as we are interested in the additional amount of memory we need in version 1 and version 2, when compared to vanilla.
  • The max memory consumption at runtime, not regarding potential /swap memory.
  • We are only interested running on a Linux operating system

Eventually, we used the rusage feature. The returned struct offers different fields related to memory. We found that for our use case, the correct value was to use the maximum resident set size (max RSS). This proved to be reliable and reasonable compared to manual calculations of the memory we assumed we require. An example code is given below.

#include <sys/resource.h>
#include <stdio.h>
#include "mpi.h"

/* Needs to be called at the end of the process */
int MPI_Finalize() {
  int rank;
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
  // We assume only MPI root should output memory consumption
  if (rank == 0) {
    struct rusage r;
    getrusage(RUSAGE_SELF, &r);
    printf("MAX RSS: %ld\n", r.ru_maxrss);
  }
  // ... Call to PMPI for actual MPI_Finalize
}