Testing in Open Source Orbiter

DarkWanderer

Active member
Orbiter Contributor
Donator
Joined
Apr 27, 2008
Messages
213
Reaction score
83
Points
43
Location
Moscow
I am trying to add tests to Orbiter public repository by using CMake and GoogleTest. GTest download, build and Visual Studio integration all work correctly - however, I'm at impasse with handling Orbiter's own dependencies. As soon as I add anything from OrbiterSDK into test code, test project fails on startup with:

orbiter.exe.problem.png
I recall this was always the case - however, this time I'm really interested in digging to the root cause of this. Does anyone have any ideas which may be helpful?
 

DarkWanderer

Active member
Orbiter Contributor
Donator
Joined
Apr 27, 2008
Messages
213
Reaction score
83
Points
43
Location
Moscow
You are right - I have tried to substitute Orbiter_NG.exe for Orbiter.exe - forgetting that NG is a thin wrapper, not the full binary. Copying correct executable to root does the trick - thanks!

A couple more questions (while I have your attention):
  • Do you have any objections if I introduce Google Test framework, or would you rather have "plain" CMake tests (or some other framework)?
  • Do you see any issue if I add a step to copy "server" Orbiter.exe to root directory, if graphics version has not been built? This will only affect "build" intermediate directory, not "install" target
 

martins

Orbiter Founder
Orbiter Founder
Joined
Mar 31, 2008
Messages
2,448
Reaction score
462
Points
83
Website
orbit.medphys.ucl.ac.uk
You are right - I have tried to substitute Orbiter_NG.exe for Orbiter.exe - forgetting that NG is a thin wrapper, not the full binary. Copying correct executable to root does the trick - thanks!

A couple more questions (while I have your attention):
  • Do you have any objections if I introduce Google Test framework, or would you rather have "plain" CMake tests (or some other framework)?
I have to admit that I am not really familiar with either, but I wouldn't mind seeing how it works - what about setting up a Google Test framework but keeping it in a separate branch until I get an idea of the mechanism.
  • Do you see any issue if I add a step to copy "server" Orbiter.exe to root directory, if graphics version has not been built? This will only affect "build" intermediate directory, not "install" target
No, I don't see a problem with that.
 

DarkWanderer

Active member
Orbiter Contributor
Donator
Joined
Apr 27, 2008
Messages
213
Reaction score
83
Points
43
Location
Moscow
I have to admit that I am not really familiar with either, but I wouldn't mind seeing how it works - what about setting up a Google Test framework but keeping it in a separate branch until I get an idea of the mechanism.
Sure - I've created 'gtest' branch on my fork - if you add it as remote to your repository, you should be able to check it out (in both senses)

I've added 2 trivial tests there to try it out - there would be 2 ways to run them:
  1. Via Visual Studio - if you open Orbiter source directory, configure & build - tests will be available in "Test Explorer" (see screenshots). Would probably work with generated solution too but I haven't tested it
  2. By running 'ctest' command in CMake build directory
Tests definition:
gtest2.JPG
How they look in Test Explorer:
gtest1.JPG
There is some instability at the moment but I think it showcases the concept well. GTest framework gives us:
  • well-defined atomic tests (which can be easily rerun individually
  • rich set of assert macros
CMake tests (add_test) are also integrated with Visual Studio but the granularity will be much coarser (will not be able to launch individual tests from test suite, only whole executable)
 

DarkWanderer

Active member
Orbiter Contributor
Donator
Joined
Apr 27, 2008
Messages
213
Reaction score
83
Points
43
Location
Moscow
Still working on adding tests. GTest proved to be too finicky ultimately - they add 10-20 seconds to "configure" time and do not preserve working directory, which randomly break test discovery. I've switched effort to push Catch2 framework instead. In addition, trying to add capability to run integration tests - a scenario with a script which should exit Orbiter with a success/error code (repurposing code which someone else committed)

Current issues:
  • Scenario test fails right now with a 0xc0000409 "status_stack_buffer_overrun" code - trying to figure out why
  • At the moment it's hard to do meaningful unit testing as most code depends on Orbiter object
 
Last edited:
Top