The case for empirical survey-based research

I have worked in research for a decade and a half, and I am privileged to have been a part of different research communities during this time. As a researcher, I was born in the design automation community, but I got increasingly involved in the real-time community as I grew up. This was very interesting for a young researcher and I learned a lot from this, because each community has its own perspective, practices, strengths, and weaknesses.

Once I joined ESI (TNO) back in 2016 and started working in applied research, I started reading publications and attending conferences in the modelling community. There, I came across empirical survey-based research that systematically investigated industry trends, needs and practices, and that studied adoption and perceived benefits and drawbacks of different technologies and methodologies. I immediately found this line of work incredibly useful as it elevated my understanding of what happened in industry from a collection of anecdotes collected over many years based on conversations with people in a few companies to a coherent snapshot of the experience of hundreds of people across industrial domains. This was very helpful as it allowed me to get a perspective on how the research that was conducted in the field fit into the industrial reality of the men and women that are ultimately supposed to use the results. From an academic perspective, I also had the feeling that this line of work provided all the citations I needed for the introductions of my publications, as it helped me position my own work on modelling in a broader industrial reality.

Empirical survey-based research is an established research direction in social science, but also in technical fields, such as software engineering and to a lesser extent system engineering. However, there was no work like this in the area of real-time systems. I decided to change this and pitched the idea to Rob Davis, Mitra Nasri, and Geoffrey Nelissen and Sebastian Altmeyer during a meeting in Amsterdam in May 2019. To my delight and surprise, they all agreed this was an important research topic and enthusiastically decided to work on this together, effective immediately.

Performing survey-based research may seem deceptively simple at first. After all, all you have to do is to make a survey with questions, ask people to fill it out, and analyze the results. Easy-peasy! However, it turns out each of these steps involve substantial challenges to ensure validity and quality of the final results. Luckily, there are well-established methods that provide guidance along the way.

We had long discussions to precisely define the objectives for our work, to identify the target population and decide how to invite a sufficiently large representative sample from it to take the survey without accidentally including people with the wrong expertise. Of course, we also needed to select the right questions to ask and decide how to phrase them to avoid misunderstandings. The latter is surprisingly difficult when the questions are sent to people with different backgrounds, working in different domains with their own definitions and jargon. Making survey questions hence requires good focus and a fine balance between being precise and concise. Too many or too long questions means it takes longer to fill out that survey, causing participants to get demotivated and drop out before reaching the end. Results had to be collected and analyzed, and the right statistical methods and tools selected to allow the survey results to be generalized to the wider target population that our sample represented. This involved a lot of work, but we did it with pleasure as we always had the feeling this was some of the most important research we had worked on.

It took us little over a year and a half from the pitch in Amsterdam until our conference paper “An Empirical Survey-based Study into Industry Practice in Real-time Systems” was published at the 41st IEEE Real-Time Systems Symposium (RTSS). The goal of this paper was to inform both academics and practitioners about the characteristics of the real-time systems being developed today and about important trends for the future, thereby helping to avoid divergence between industry practice and fundamental academic research. To this end, the paper presented results and observations from our survey of 120 industry practitioners in the field of real-time embedded systems. A separate session was dedicated to this work on the last day of RTSS 2020. The session began with a 25-minute paper presentation, which is available here:

The paper presentation was followed by a panel discussion involving three industry practitioners from the three main industrial domains covered by the survey: Marcelo Lopez Ruiz (Microsoft), representing the consumer electronics industry, Simon Schliecker (Volkswagen), representing automotive, and Stephen Law (Rolls-Royce), providing an avionics perspective. The panel discussed four key questions relating to the survey results, including what characteristics of real-time systems are the most relevant with respect to their domains, and what are the most relevant trends in real-time systems development now, and looking ahead over the next decade. The panelists concluded by giving recommendations about future research to the academic community. The opening statements from the panelists related to the four questions was pre-recorded and followed by a live discussion. The pre-recorded part of the panel is available here:

In the year following the publication of our paper in RTSS, we continued with the next steps in our research. This effort recently resulted in an extended version of our paper entitled “A Comprehensive Survey of Industry Practice in Real-Time Systems“ being published in the Journal of Real-time Systems. The main extensions in the article include a statistical analysis that generalizes the results from the survey participants to the wider population they represent. It also contains a discussion of the results of a quiz aimed at determining if the findings of the survey are common knowledge in the real-time systems community, a litmus test to see if the survey results teaches the community something new. In addition, more data from the survey has been released, allowing interested readers to further delve into the similarities and differences between the avionics, automotive, and consumer electronic domains.

So, what did we actually learn from our two-and-a-half-year venture into empirical research into industry practice in real-time systems, apart from applying new research methods? Below, you find the six main conclusions from our research. The evidence supporting these findings are available in the journal article mentioned above.

1. The survey results show that industry recognizes the importance of timing predictability, but that other design aspects are of equal or greater importance, such as functional correctness and reliability/availability. Hence, it is important for real-time systems research to be cognizant of its impact on these aspects.

2. Many real-time systems today are distributed systems that use multi-core processors, and have complex memory hierarchies. Further, multiple different operating systems and networking technologies are typically utilized within the same system, as are different types of timing constraints and task activation patterns.

3. Many respondents did not consider timing constraints to be sacrosanct, with even the most time-critical functions allowed to miss some deadlines.

4. There is no silver bullet to manage timing behavior in complex real-time systems. Instead, the survey reveals that a wide range of different tools, techniques, and policies are used for timing analysis, scheduling, and to increase timing predictability. There is no one size fits all solution.

5. The trends suggest that single-core systems are still widely used today, and are expected to remain relevant for new developments for at least the next ten years. However, more complex (heterogeneous) multi- and many-core systems are already prevalent, and their adoption is expected to increase significantly during the 2020s.

6. The results of the quiz show that the aggregate findings of the survey are not common knowledge among industry practitioners or academic researchers. Hence, the survey fulfills a very important and necessary purpose, to help bring members of the real-time systems community up to date in terms of their aggregate view of current industry practice in real-time systems design and development.

Next Steps

I hope that our work inspires others to conduct empirical research in real-time systems. There are many ways to continue with this line of work. First of all, others need to replicate our results to validate that they hold for different samples of the target population. For this purpose, we are happy to transfer the survey we made on SurveyMonkey, such that it can be reused. Secondly, our survey was very broad and covers real-time systems across many application domains. More specific questions could be obtained if the focus was on a single domain, although the main challenge will be finding enough representative participants with a narrow focus to get reliable results. Thirdly, surveys are only one way of conducting empirical research. Another method sometimes used in software engineering is to use interviews, allowing more in-depth questions to be asked. However, the drawback of this method is that it is more time consuming to interview a large number of participants and to encode and analyze the results. Ultimately, it is not necessarily a matter of choosing whether to use surveys or interviews, as they are complementary research methods and can strengthen each other when combined.

Empirical research is an emerging research direction in real-time systems research, and we hope it will grow and become a well-established part of the research conducted in the community. This would help us better understand the industry we are trying to serve and help us close the gap between academic research and industry practice. A first important step is that empirical research is recognized by all main conferences and journals in the area of real-time systems and explicitly included in the call for papers. You can play an important part here by helping us communicate the value of empirical research to others in our community and beyond. Triggered by our research, the Technical Committee on Real-time Systems (TCRTS) has already extended the scope of the conferences it sponsors to include empirical research, which is a good start. It has also made a form where industry practitioners can register as TCRTS Industry Contacts.

Contacts on this list can opt-in to be contacted for survey-based studies, thereby contributing to making future studies easier to conduct and more reliable. Thank you for signing up and stimulate future empirical research!

An earlier version of the post has appeared on the SIGBED blog.