I lied!
We’re not talking about Audio this week, we’re talking about Quality Assurance, otherwise just known as QA! Excuse the dust as we teardown and set up this new development diary a little earlier than planned. We will eventually have an Audio Dev Diary (looks hopefully at Community) but for now you’ve all locked in here with me again about a subject I know all too well: how it is to QA Victoria 3.
Though this is honestly a great introduction to what it's like to QA Victoria 3, or how it is to be a QA in general. As we are at the end of the pipeline we must be the most adaptable members of the team. Broken builds, delays, reworks, and merge errors - these are an everyday occurrence in our profession. Even when things go perfectly, QA is still a race against time or more specifically a race of timeboxing: what is the most important thing to test now and what can be put aside to when there is either time or it is more complete - do we suspect another Market Rework is coming or are we at the final implementation at last?
A regular QA posting, usually among ourselves.
Not everything can get tested, it's a sad fact. Paradox games are unique in their complexity of interlocking mechanics and systems that to test the full functionality of the game and all of its iterations for every change is impossible. An infinite number of QA with an infinite budget and time could still not deliver a near perfect quality product. Why is that? Well because in that scenario the bottleneck of quality would not be the QA but the developers on the other side of the bug fixes - it's their bandwidth that would then be the limiting factor. The Job of QA is to test and test appropriately balancing the time for technical feedback and gameplay balance based on the development timeline of the game.
And before we get too far, the comment that I expect to get the most to this diary: “Why don’t you use [insert specific development/testing process], it will solve all your problems!”
I wish this were true, but there is no silver bullet for development.
Each model of development and testing mitigates risk in a separate way, but there is still inherently some risk especially because we are talking about the potential for human error: missing connections or building on top of a system that was not originally written with such a perspective multiple iterations ago.
The most lock-tight development model that does not allow for any “bugs” also does not allow for any creativity on the part of the developers. Video games may be a product to be sold but they are also inherently an art, which makes QA’ing and solving their problems a much more subjective experience at best. The trick is to find a system that creates as many safeguards as possible without fully boxing in the creativity of the team. I digress and will stop there before I get into a longer wided sidenote. Throughout this diary I will go into more detail about what I mean.
The People, The Process, The Timeline, and Going Forward
As we’ve moved through the more public side of the development process, my role in QA has shifted while the meme of me as a QA Lead has remained the same. I’ve gotten a lot of questions about “what exactly a QA Lead does and how my role as Manager/Director was different” so I thought I would take some time to flesh this out for you all with at least a basic explanation.
The People - QA Team Breakdown:
QA Director: Represents the QA team’s interests on the studio leadership level, ensuring that all plans of release and development are taking into consideration the bandwidth and requirements of their team.
QA Manager: Professionally manages the QA Team, less involved in the day to day and more involved in the growth and development of the QA Team as individuals and overseeing staffing.
QA Lead: Represents the QA team’s interests on the project leadership level, manages day to day prioritization and coordination of testers.
QA Tester (Embedded): Employee of Paradox Development Studio, who lead the charge in testing development, verifying fixes, and giving feedback.
QA Tester (Outsourced): Employee of our Outsourcing Partners (such as QLOC) who works with the internal team and assists in testing development, verifying fixes, and giving feedback.
Community/Beta Tester: Paradox Community Member who under NDA gets access to an in-progress build to give feedback and partial bug reports on - which the QA will later collate and verify.
Overall QA are an interesting bunch of people. It takes a very dedicated individual to look at an unfinished project and happily get to work, to not only document the flaws but to peer behind the curtain and see the intent of the finished product and give feedback on that. The QA, as an individual, is able to look a developer in the eye, tell them they love a feature and then send them 20+ jiras about all the problems with it with no cognitive dissonance. Sometimes we will throw in a few feedback jiras about how we think it would be cool to tie it into other features as well.
Checking if the issues are actually fixed is always an adventure - you never know which new issues you're gonna find.
Our need to be critical and take the player perspective also leads to us being very blunt and forward when asked our opinions. In our profession, sugar coating concerns only lead to problems down the line. The trick that I will share on the forums is this - it's possible to be blunt and not a jerk. Browbeating a developer as a QA may get the problem solved but it comes at the cost of the working relationship. I have been many types of QA in my career and if I may be so bold to reflect upon it - it is the QA that is blunt yet respectful that gets more achieved in influencing the game then the one who screams the loudest. As developers are people too, they have feelings and can react emotionally especially if you act emotionally (especially angry) towards them. I ask that you please remember this as we move towards post launch and live support, but I will get into that more later in this dev diary.
We were once told that we should go about things in a more positive perspective. That request did not stick for long.
I’ve joked on stream that “I saw the QA team laughing during multiplayer testing and that’s either a sign that everythings working or everything is broken.” Our profession comes with a strange sense of humor, or maybe it's just the type of people who enjoy our work - after 8 years I’m still not sure.
The Process
I’m going to talk more about the general process of QA and try to avoid going into too much detail here, otherwise it is a rabbit hole I will never escape. What I will say is this, I am talking high level processes of the QA department as a whole in relation to the project, QA’s interfacing with each other department: Code, Design, Narrative Content, VFX, Audio, UI/UX, Environment Art, Tech Art, 2D Art and the various subcategories I’ve forgotten - each of these have a separate process work a bit differently. This is because they are all developed differently and implemented at different times of the game and so the standard QA has for one team is different from another depending upon the state of development.
QA has two types of general process testing, Milestone and Feature.
Milestone testing is the one most people outside of the industry understand as it translates to the words we use - Proof of Concept, Alpha, Beta, Release Candidate, etc. These are all milestones that we use in the development process. The intention is that there is a significant difference between them, the status of its features and even more importantly its gameplay loops and engagement. Milestone encompasses the game as a whole and is generally used in the statements such as “its as you would expect for alpha” etc.
Even while testing the features for milestones. QA, just like the rest of the team finds a way to add a little humor into the documentation.
Feature testing is a bit different, here at Paradox we usually refer to it as “development support testing” it's where QA is doing more piecemeal testing of individual implementation of the various pieces of the game. This is where we are ensuring that things merged correctly, that coders did not accidentally overwrite each other and we are caring more that things are working from the technical perspective then the full gameplay perspective.
When a game is in its early stages of development to its alpha milestones - feature testing supersedes milestone testing. That’s not to say we do not do alpha milestones but a large part of our testing is in the technical ends of the game as opposed to “playing for balance purposes.” The reason behind this is that while the game is still actively being developed and is not yet towards the milestone where features are less likely to change, testing balance is inefficient effort. QA has a limited amount of time and a never ending series of tasks to tackle, so we have to be as smart as we can.
Early on the fact the game is less polished has to be accepted by the QA team. We find unique ways to cope with this fact.
In case you were wondering, this is what that smile looks like today.
As we get further along our milestones, towards beta and release candidates - the inverse is true: milestone testing supersedes feature testing as the main means of gathering data. In beta QA may still dive into features and test them from a technical perspective depending upon the number of changes made by a developer but more often than not they will focus on how these features interact with each other and do they make for meaningful and fun gameplay. Our difficulty in doing this? We need to figure out how to answer this question with some form of certainty as quickly as possible.
As the game’s development progresses so too does QA’s standard processes on what is worth bugging and what is an acceptable fyi issue. Technical concerns have priority in early stages of milestone development while in release candidate papercuts with poor UI or incorrect information given to the player are treated as the same severity as broken functionality outright. Because it doesn’t matter if it technically works if the player cannot understand it.
The Timeline Going Forward (and Farewell)
A QA’s role adapts as the project progresses. In the beginnings of the project we’re very technically minded and working on ensuring the structure of the game is there. As development progresses we focus more on the features and how they feel and the overarching scope of the game. As we get further along the milestones QA becomes the representative experts of their game. Designers can tell you how it's supposed to work, QA tells you how it does currently and because of this knowledge we work regularly with community, beta testers and influencers in teaching them how things work early on and getting their feedback. Community is very much our comrades in arms in this endeavor, anything reported to Community Team on our forums or discord is brought to QA’s attention to ensure we’ve got it Jira’d and the rest of the team is aware. Soon release will be upon us, an exciting time for us all. You all will finally get a hold of the game and the development team will sit here anxiously waiting to hear your feedback and see the shenanigans you all get up to. The QA team itself is going to sit here wondering what last minute changes we made broke what and how that slipped through the cracks of our testing. At release and going forward we will have a bug forum where any issues found by you all that you think we do not know about, you can let us know. And even if we do know about it, you can help judge priority of fixing but more information will come in regards to that from the QA Team Lead around release so keep your eyes peeled. And well, that’s a summary of high level QA processes, instead of talking about the nitty gritty of testing X features or how many jira’s we’ve written I hope you enjoy the peak behind the curtain to our sense of humor splattered throughout the dev diary. I guess this dev diary also marks the end of my responsibilities as QA on the team. I now go off to the other side of the development… ready to take on a different kind of responsibility. Be kind to my QA Team in the coming weeks. You have no idea how much effort they have put into this game over its development - especially in the past year. I certainly do, and I’m very proud of them. The project is in good hands. Hopefully next week we’ll have our Audio Development Diary with Franco, sorry I’ve got no classy segue this time.