Anthony Rogers Interview (QA Lead on Dead Money)

NF: Can you explain what your responsibilities were as QA Lead on Dead Money? Did these change from your time on the Fallout New Vegas base game? At what point did you join the Dead Money team, was this during the final months of Fallout: New Vegas’ development or after its completion?

AR: As QA Lead on Dead Money and all the DLCs, I oversaw the small in-house QA team, only 3 people including me. I managed our Bug Database, assigning bugs to the development teams and directing questions from the development teams about bugs to Bethesda's QA department or my team depending on one who had written the bug in question.

The in-house team was mostly used by the developers to test new features or areas as they were being created to provide feedback and find issues, as we were able to grab updates from the developers much quicker than we could provide those same updates to the Bethesda QA staff.

We also continued to test Patch content in the base game as those fixes were found. I was made QA Lead on Dead Money, the QA Lead from the base game [Fallout New Vegas] became a Producer so I took over his duties on the DLC. I started on the DLC in the final months of the base game's development, I don't remember the exact timing of it all as I was wrapping up the base game and familiarizing myself with the design and story for Dead Money as it was coming online.

NF: Could you give a brief summary of your time in the industry up until the point at which you worked on New Vegas and Dead Money? How did you come to work on these specific projects?

AR: I started out in the Industry working as QA at Activision on Enemy Territory: Quake Wars. I was there for about 4 months before moving to Obsidian to work QA on Neverwinter Nights II: Mask Of the Betrayer. After that expansion wrap up I was moved on to QA for Alpha Protocol as the Second Expansion Storm of Zehir [Expansion for Neverwinter Nights II] was moving forward with a smaller team and the Alpha Protocol team was ramping up into full production. Once Alpha Protocol finished, I was moved onto Fallout: New Vegas, as it was entering its Alpha development and needed more QA staff to support it.

NF: What were your hours like during the project? Were there ever periods where those hours changed? That can apply for the entirety of the development of the Fallout New Vegas DLC expansions given how long ago it was.

AR: Throughout the development I averaged a normal 40 hours a week. There were periods close to deadlines where that could go up to 50-55 hours a week, but Obsidian tries not to mandate overtime and does not tend to mandate working weekends. There was some crunch that occurred but not in the high levels you may have heard from other studios and not for as long.

NF: What was an average day of production like? What did meetings look like? What form did communication between the QA department and the rest of the production staff take? Did the workflow of identifying bugs and recreating them change on Dead Money versus the base game?

AR: On an average day I would come into work and begin grabbing the latest build of the game and check emails. While the download [of the build] was going, I would open the Bug Database and assign out any bugs that had come in from the Bethesda QA team, as they start 3 hours before I did from being on the East Coast.

I would also assign any issues that were marked as “Need More Information” out to the QA staff that had written the bugs so that the person with the most understanding could respond and help clear up any misunderstandings or provide the additional information requested by the development team. I would also update the team with the bug tracking numbers to show how many new bugs were opened and closed from the previous day. After the download finished, I would load up the game and play through the beginning of the game to make sure everything was working correctly.

I would also run through the beginning on the DLC from the beginning to make sure the transition and DLC activation was working correctly. If there were meetings that day, I would sit in on them and listen to what updates had been made to areas and what features were in progress. If there was something that needed QA attention they would be brought up in these meetings or through Email, so that I could assign out the task to my team. If there was nothing specific that needed to be tested, I would spend the day just playing the game and trying to find any issues I could anywhere in the game. I would also periodically load back into the Bug database and assign out bugs as needed. Occasionally a developer would email me or come to my office and request something be tested before they would check it in, by providing me with a custom file with their changes.

This did not change as we moved from the base game to the DLC. We typically had a weekly team meeting that covered the overall production of the game and important deadlines coming up. Then the smaller strike team met for the smaller features and areas to update progress and address issues specific to those areas or features as they were being developed.

NF: Were there any specific challenges with Dead Money or Fallout: New Vegas in general that made testing different from other games you had worked on?

AR: There were no specific challenges I encountered with testing FNV or Dead Money. Since we were using Bethesda's engine and tools the process for testing the game was pretty smooth as it was easy to get new builds each day and it was easy to test changes for developers to help them ensure that they did not add bugs or features that would cause issues for the rest of the team.

NF: Do you remember any specific bugs or areas that had a myriad of bugs associated with it? Did the new mechanics like say radios triggering the bomb collars prove more or less problematic for testing purposes?

AR: I can't remember anything specific that had many issues as we had console commands and things we could easily activate to get around issues as they came up. And depending on what we were testing we could use those [console] commands to bypass new systems that might not have been working correctly on a given day or were still in progress.

The only issue I remember specifically from Dead Money was trying to help design come up with a way to limit the amount of gold bars you could level the DLC with, as it was sort of the moral of the DLC. They wanted to make them something the player would want to leave with and saw value in, but the different inventory systems and ways you could move things around in the world made it difficult to achieve that goal, as some players are stubborn and would try to take everything, but doing so would affect the economy of the game as they were worth so many caps.

NF: During the creation of Dead Money, were you splitting your time between Dead Money and the other DLC projects such as Honest Hearts and Old World Blues as they started their productions? If so, how did this affect your team's ability to identify bugs and recreate them?

AR: The DLC was staggered in a way that one was ramping up and the next was ramping down. Towards the end of Dead Money, I was splitting my time with Honest Hearts. Since Honest Hearts was early in production there was not as much to test, so by the time Dead Money was finished, we just rolled the QA team onto the next DLC. Also, since the DLC's were self-contained, there was less likelihood that one DLC would cause issues in the other.

NF: Given how long ago all of this was, I wanted to ask some more generalized questions regarding your time in QA to get more of an average of your experiences.

How did you get into QA originally, was this your first role in the industry?

AR: As mentioned earlier I started in QA at Activision after graduating college. I was looking to get into the industry and at the time, 2007, it was easier to get your foot in the door starting in QA than it was to find a position as a designer, artist, animator, etc.

NF: If someone had no knowledge of what function QA provides to a project, how would you describe their role in a games production?

AR: QA is the front line for finding issues with a game and providing proper steps to reproduce those issues so that the development teams understand and fix those issues. Good QA can see an issue and provide clear directions to the development team so that they understand what the issue is and how to make it happen again. So, that the person working to fix the issue can reproduce the issues themselves so that they can then provide a proper fix. Those clear steps are also needed by the QA team to then verify that the fix is correct since it will not always be the person who found the issue that is verifying the fix for the issue. One misconception people who have some knowledge of QA is that QA oversees fixing the issues they find. This is the case in other tech industries, but not in game development.

NF: Could you describe what an average milestone or sprint would look like for a QA team?

AR: Usually, the start of any milestone is spent just looking for general issues in the game as the new features or areas are being created, as there are no specific changes in the build to test. As time goes on QA will start to test the features as they start to come online. The end of the milestone is typically when extra hours may be spent to test the features to assess as many issues that have been found and can be fixed for features that are required for the milestone. This is also the time that QA is verifying the fixes that have gone in throughout the milestone.

NF: Did you see yourself on equal terms with the rest of the production staff positions when working in QA? More specifically, was there ever a time when working in QA where you felt your position was not taken as seriously as say a programmer or designer? (I wanted to ask this because of how QA teams have described their experiences at other studios).

AR: There was never really a time where I felt that the other staff did not take me or my position in QA seriously. Most of my time in QA, I was working in the studio and had face to face contact with the other developers, so any time they needed help understanding an issue I wrote up or needed help reproducing an issue they could just talk to me. Sometimes at some studios QA gets seen as these faceless names attached to bugs, and the only communication between the developers and QA is comments on bugs.

NF: How would you describe your time in QA? Would you describe it as a positive or negative experience?

AR: My time in QA was mostly positive. I worked with a lot of great people and made a lot of friends throughout the years. Most of the negatives more came from the typical game development issues and experiences of having games cancelled or seeing friends lose their jobs because of cancellations.

NF: In closing, if there are any stories you’d like to share from the development of Dead Money or Fallout New Vegas overall, please feel free to share them here.

AR: The only story I remember is from the launch of the Base game, where Doc Mitchell's head would spin around on PC when you first load into the game. People tend to blame this on there being bad QA or that we were incompetent, when in fact this turned out to be some data corruption on the Steam depot for the Live game. This was tested every day by multiple people on every platform so it is not something that would be missed, but through systems not in control by the QA department an issue appeared that lots of players saw and pointed the blame at us. One thing that is always forgotten in terms of game Development and QA is that in the first hour of a game being out, more man hours will be put into playing the game than the entire time of the game development. It is highly likely that players can find something that was missed by QA. If 100,000 players play the game in that first hour that ends up being 100,000 man-hours, or 11 years worth of play time in comparison to the 18 months that FNV was in production.