Ranking system with two score variables

dgatsoulis

ele2png user
Donator
Joined
Dec 2, 2009
Messages
1,924
Reaction score
340
Points
98
Location
Sparta
The ranking system on a challenge that I am making has two criteria: Delta-V used and Time of Flight. Obviously when two players have used the same Delta-V, the one with the higher rank is the one with the less TOF. And of course, when 2 players have the same TOF, the one with the less Delta-V gets the higher rank.

But how do I compare and rank accordingly, players that have different Delta-V usage and TOF?

For example:

We start with a budget of 3000 m/s and 150 days to complete the challenge. If either of those two variables "runs out" before the challenge goal has been met, the player gets disqualified.

I have already calculated the "perfect" score for each variable individually: Let's say, for the maximum TOF (150 days), the best you can do is 2700 m/s in Delta-V and for the maximum Delta-V (3000 m/s) the best that can do is 130 days in TOF.

So two players that have the following scores:
Player1 DeltaV 3000 m/s TOF 130 days
Player2 DeltaV 2700 m/s TOF 150 days

are both ranked as equals with a perfect score.

How would I go about to compare players with results "inbetween"?

If one has DeltaV 2800 and TOF 140 and the other one has DeltaV 2750 and TOF 145 days, which one has done better than the other?
 

BLANDCorporatio

New member
Joined
May 29, 2013
Messages
67
Reaction score
0
Points
0
The general answer is there is no general answer.

You have two numbers. Somehow, you want to crunch them into a single figure of merit. Some information will be lost (let's not get into ways to fill planes with a curve now they're not relevant ;)). You might prioritize one figure of merit over the other by weighing it more, for example.

In particular, one ranking that makes sense is TOF first, deltaV second. In detail:

- if deltaV budget exceeded, disqualify
- rank according to TOF
- break ties by deltaV

This seems "reasonable" because if you have a given budget for deltaV ... then you have that budget for delta-V :p Presumably there's no gain in completing the mission with tanks half full, but rather there is a premium on getting there faster. The mission is about getting there fast, isn't it?

But maybe it isn't. Then, alternatively, you may:

- if time budget exceeded, disqualify
- rank according to delta-V
- break ties by TOF

This also makes sense, but the 'meaning' here changes. The idea is not just to get from A to B, it's to get there with as much propellant as you can. If that's your mission, then this second ranking is reasonable.

So basically ... what is the mission? That's what determines what you prioritize.
 

MattBaker

New member
Joined
Jul 9, 2011
Messages
2,750
Reaction score
0
Points
0
Well, I would do percentages. You know the ideal case, that's a 100%. Then you have the worst allowed, 0%.

In your case 130 days is the best, 150 the worst. So if I take 135 days I achieved 75%, in 140 days 50% etc.
Same with delta-V: 2700 m/s is the best, 3000 m/s the worst. If I take 2800 m/s I am only at 67%.

So in the end you have two percentages which get added up and the one with the highest overall wins. To answer your example:
2800 m/s: 67%
140 days: 50%
Overall result: 117 points

2750 m/s: 83%
145 days: 25%
Overall result: 108 points

And if you use a detailed time of flight with hours or minutes and a decimal in delta-V even closes cases could be decided by that.

Although for this system to work you need to know the ideal case, if someone comes up with a method to use less delta-V or be a bit faster it collapses. But then again: Doing stuff the developer (challenger) didn't know always tends to result in weirdness.
 

dgatsoulis

ele2png user
Donator
Joined
Dec 2, 2009
Messages
1,924
Reaction score
340
Points
98
Location
Sparta
Thank you both for your input.

@BLANDCorporatio, yeah I figured as much when I run into this. A priority needs to be given on one variable over the other, considering the mission aspects.

But in this case the whole point of the challenge is to teach the player about the delta-V vs TOF aspect of spaceflight and the trade-off between those two.
But as I suspected and you pointed out, there seems to be no objective way to measure both at the same time.

@MattBaker

That's almost exactly the way I thought of doing it, with the difference that the starting point was 50% for a flight just within the budget (3000 m/s and 150 days).
But the numbers don't work out, since a "perfect" flight can be either 3000 m/s 130 days or 2700 m/s 150 days.

These two get a "perfect" maximum score.

With this scoring system someone who made the flight in 2750 m/s 140 days gets a bigger score than the perfect flight.

But... :hmm: now that I think about, it could work if I was subtracting the percentages from 100 first and then added the points together.

I'll setup a spreadsheet to see how it works out.

Thanks again for the great input. If in the meantime someone comes up with a better idea, I'd love to hear it.
 

MikeB

Member
Joined
Feb 25, 2009
Messages
185
Reaction score
0
Points
16
Location
Seattle
The SPECmark approach to benchmarking computer systems used something like the following:

Given N measured quantities and their corresponding 'standards', make N ratios, e.g., measured/standard1, measured/standard2, etc. In your case, N is 2 for DV and TOF.

Multiply the ratios, and take the Nth root (i.e., geometric mean) of the product; square root for N=2. That's the figure of merit for ranking.

Make sure to be consistent about the way ratios are made, so that smaller is always better, or larger is always better, but don't mix them.

I'll do the math on your examples when I get home.

------ OK, here are the calculations for the examples ------
Code:
player   TOF   DV      Ratios     Prod   Sqrt
-------  ---  ----  ------------  -----  -----
Stndrd   150  3000							
Player1  130  3000  0.867  1.000  0.867  0.931
Player2  150  2700  1.000  0.900  0.900  0.949

For this example, Player2 has score of 0.949; Player1 has 0.931. Since the ratios in this example are set up so that smaller (shorter TOF or lower DV) is better, Player1's score is better.

I hope I did that right!
 
Last edited:

dgatsoulis

ele2png user
Donator
Joined
Dec 2, 2009
Messages
1,924
Reaction score
340
Points
98
Location
Sparta
Thanks a lot Mike for the method and the example. My numbers work out the same too. The main problem is that I need to "equalize" the results of the 2 players with the best score in dV and TOF respectively and rank them both as equal scores.

This indicates that the TOF - DV relationship is not linear, which was my main (wrong) assumption.

I'll need to use more data from several actual flights and pin down the DV- TOF relationship, which of course will only work in this particular flight.
 
Last edited:

Thorsten

Active member
Joined
Dec 7, 2013
Messages
785
Reaction score
56
Points
43
DeltaV and TOF are highly non-linear. For example, say you want to go to the moon. As you climb out of Earth gravity field and shed your ejection velocity gradually, there will come the point where the pull from Earth equals the pull from Moon, and that'll be the slowest point of your trajectory.

If your initial ejection velocity was such that you reach this point with just zero velocity, you'd be sitting there infinitely long if the Moon wouldn't move. If you'd reach it a bit faster, the Moon would soon accelerate you and it'd take you much less time. So a tiny DeltaV can change TOF a lot when you're operating close to such a cusp.

A somewhat 'fair' solution to the problem you're discussing here requires you to know the optimal trajectory. For instance, fix the DeltaV budget. If everything runs optimal, there's a theoretically fastest trajectory for this budget (which you can compute by numerically going through a bunch of different trajectories and an optimizing tool). You can then measure how good the player performed on the actual trajectory against what would have been possible.

Say for 2700 m/s the best TOF is 150 days and someone took 154 days, then he's 2.6% above optimum. Say for 3000 m/s the best TOF is 130 days and someone takes 134 days, he's 3% above optimum, so he's been doing worse than in the first case. If you know what for each Delta V the best possible TOF is, you have a meaninful comparison standard how well people did.

Of course the potential catch is that the problem is non-linear. So I assume you want to measure skill in some way. And for that trajectory stability matters.

For instance, you may time a burn by 1% wrong. If you're close to a cusp, that might in the end influence your TOF by 30%, but if you're away from the cusp, the TOF might just be different by 2%. So if the skill you want to test is for instance accuracy in maneuvering, then this matters, someone aiming to fly a trajectory close to a cusp has a much harder time staying close to the optimum than someone away - even a tiny mistake will immediately incur a penalty.

But of course you can always argue that comprehension of the problem is part of the challenge, and people should figure out themselves what trajectory they should fly or not, given the available options :)

So I think there is a solution, but it really requires you to know the theoretical optimum with reasonable precision.
 

dgatsoulis

ele2png user
Donator
Joined
Dec 2, 2009
Messages
1,924
Reaction score
340
Points
98
Location
Sparta
I graphed it so I could have a visual representation of the scoring system.
Graphic1_zps594a02ad.jpg


As you can see I drew 3 different graphs each representing a different "best result" line.

The actual shape of the line will be determined by making the flight with a few dV samples and optimizing the TOF for each one.


So, after I have determined the area of possible outcomes, what's the comparison between two valid scores? The shortest distance to the best results line?

For example I have two scores A and B inside the area of possible outcomes.
Score A will have a shortest distance A' from the line and score B will have a shortest distance B' from the line.
If A'<B' then A' is a better score than B', correct?

EDIT: I guess this is what BLAND meant with
let's not get into ways to fill planes with a curve now
 
Last edited:

Thorsten

Active member
Joined
Dec 7, 2013
Messages
785
Reaction score
56
Points
43
So, after I have determined the area of possible outcomes, what's the comparison between two valid scores? The shortest distance to the best results line?

Correct. Or you can take y-distance given the x of the actual flight, or x distance given the y of the flight.
 

dgatsoulis

ele2png user
Donator
Joined
Dec 2, 2009
Messages
1,924
Reaction score
340
Points
98
Location
Sparta
That's great. Thank you very much for your input. It will take some work to figure out the line of best results, but at least I will have an objective scoring system.

If by any chance a player performs better than expected, I can use that score as one of the new limits of the area, or give them a score above 100%.

I guess if I am ever crazy enough to use three scoring variables, all I need to do is add another axis and calculate the volume of the possible outcomes and then the area of the best results surface. The score comparison will similarly be the shortest distance from the best results area.
 
Top