Project Community based station construction and stationkeeping

Donamy

Addon Developer
Addon Developer
Donator
Beta Tester
Joined
Oct 16, 2007
Messages
6,907
Reaction score
205
Points
138
Location
Cape
Just a curious question.
 

Majid

Active member
Joined
Oct 31, 2014
Messages
156
Reaction score
27
Points
43
@Face -> One of the reasons you saw instability and crashes is because I cut corners and instead of implementing my own MJD source, I was using this: http://tycho.usno.navy.mil/cgi-bin/daterdnm.sh

And it turns out they started throttling that endpoint or something because I noticed we have started getting null responses and my code wasn't handling that.

No matter, I used this: https://www.npmjs.com/package/orbjs
and with that the endpoint for the MJD becomes really simple:

Code:
app.get('/mjd', function(req, res){
  res.status(200).send(""+orb.time.JDtoMJD(orb.time.dateToJD(new Date())));
});

Plugin updated, same link: http://orbiter.world/orbiter.online.zip
This should be much much more stable.

---------- Post added at 12:21 PM ---------- Previous post was at 12:16 AM ----------

Ran Orbiter all night. No crashes, still running, everything survived the night. What a relief ;) My DG somehow ended up miles away from where I left it, but that's on the client machine.. Kind of puzzled about that.
 
Last edited:

Face

Well-known member
Orbiter Contributor
Addon Developer
Beta Tester
Joined
Mar 18, 2008
Messages
4,403
Reaction score
581
Points
153
Location
Vienna
Ran Orbiter all night. No crashes, still running, everything survived the night. What a relief ;) My DG somehow ended up miles away from where I left it, but that's on the client machine.. Kind of puzzled about that.

:cool: I'll have some time tomorrow to get online, looking forward to test it then.

Quick question: what about proxy mechanisms? Does libcurl support them?
 

Majid

Active member
Joined
Oct 31, 2014
Messages
156
Reaction score
27
Points
43
:cool: I'll have some time tomorrow to get online, looking forward to test it then.

Quick question: what about proxy mechanisms? Does libcurl support them?

Can you be more specific? I am not sure I understand what specifically you mean by proxy mechanisms? I just abstracted libcurl into my own helper library with just 2 methods, curl_get/curl_post an use those for everything. Do you have a specific use case in mind for what you are talking about for context?
 

Face

Well-known member
Orbiter Contributor
Addon Developer
Beta Tester
Joined
Mar 18, 2008
Messages
4,403
Reaction score
581
Points
153
Location
Vienna
Can you be more specific? I am not sure I understand what specifically you mean by proxy mechanisms? I just abstracted libcurl into my own helper library with just 2 methods, curl_get/curl_post an use those for everything. Do you have a specific use case in mind for what you are talking about for context?

I mean HTTP proxy settings.

---------- Post added at 18:56 ---------- Previous post was at 17:34 ----------

Loaded the new client, synced, added new DG "Face", pressed PER, CTD. It seems like the vessel made it into the tele list, though. Any log or somesuch I can send to you?

---------- Post added at 19:01 ---------- Previous post was at 18:56 ----------

Exactly the same problem on a second run.
 

Majid

Active member
Joined
Oct 31, 2014
Messages
156
Reaction score
27
Points
43
I mean HTTP proxy settings.

---------- Post added at 18:56 ---------- Previous post was at 17:34 ----------

Loaded the new client, synced, added new DG "Face", pressed PER, CTD. It seems like the vessel made it into the tele list, though. Any log or somesuch I can send to you?

---------- Post added at 19:01 ---------- Previous post was at 18:56 ----------

Exactly the same problem on a second run.

Face, I noticed the CTD in hitting PER, too. The crash happened in libcurl, I couldn't reproduce it readily. Are you able to?

I did some research, some people say it might have to do with mismatch of CRT's. I built libcurl using VS 2015 in release x86 mode. I thought it might also have to do with threading. I am pretty certain libcurl is thread-safe. I was thinking of wrapping my helper curl methods in critical sections just to make sure. I actually had wrapped my methods in critical sections previously, but upon reading that curl is thread safe I removed them.

At any rate, I couldn't reproduce the problem consistently enough to determine the root cause.

I'll look into this when I get home and during the weekend.

---------- Post added at 05:57 PM ---------- Previous post was at 05:33 PM ----------

I logged on from work using a fresh install of Orbiter. I was able to see the vessels already on the server and I could see your new Dg's. I also added a new vessel called Atlantis.
 
Last edited:

Face

Well-known member
Orbiter Contributor
Addon Developer
Beta Tester
Joined
Mar 18, 2008
Messages
4,403
Reaction score
581
Points
153
Location
Vienna
Face, I noticed the CTD in hitting PER, too. The crash happened in libcurl, I couldn't reproduce it readily. Are you able to?

I did some research, some people say it might have to do with mismatch of CRT's. I built libcurl using VS 2015 in release x86 mode. I thought it might also have to do with threading. I am pretty certain libcurl is thread-safe. I was thinking of wrapping my helper curl methods in critical sections just to make sure. I actually had wrapped my methods in critical sections previously, but upon reading that curl is thread safe I removed them.

At any rate, I couldn't reproduce the problem consistently enough to determine the root cause.

I'll look into this when I get home and during the weekend.

---------- Post added at 05:57 PM ---------- Previous post was at 05:33 PM ----------

I logged on from work using a fresh install of Orbiter. I was able to see the vessels already on the server and I could see your new Dg's. I also added a new vessel called Atlantis.

I can reproduce it consistently. Everytime I try to persist a ScnEditor-spawn vessel, Orbiter crashes. No exception dialog, just a hard CTD.
 

Majid

Active member
Joined
Oct 31, 2014
Messages
156
Reaction score
27
Points
43
I can reproduce it consistently. Everytime I try to persist a ScnEditor-spawn vessel, Orbiter crashes. No exception dialog, just a hard CTD.

Turns out I was using libcurl wrong. I needed to initialize it before starting any other thread, which I wasn't doing. Secondly I was sharing handles across threads which is apparently a huge no no. I guess it's a testament to orbiter and libcurl that the thing ran all night doing at least 2 requests a second (1 for MJD).

Modules are updated in the same location. Aside from the libcurl changes, I have increased the MJD request interval to 2 seconds and have added some rudimentary interpolation.
 

jedidia

shoemaker without legs
Addon Developer
Joined
Mar 19, 2008
Messages
10,875
Reaction score
2,129
Points
203
Location
between the planets
It's nice to see another multiplayer endeavor that doesn't immediately go nowhere. :thumbup:
 

Face

Well-known member
Orbiter Contributor
Addon Developer
Beta Tester
Joined
Mar 18, 2008
Messages
4,403
Reaction score
581
Points
153
Location
Vienna
Modules are updated in the same location. Aside from the libcurl changes, I have increased the MJD request interval to 2 seconds and have added some rudimentary interpolation.

Now it keeps on running when I press PER. However, if I log off and log on again with the same persister ID, my vessel is not synchronized back again. If I log on after deleting the ID file, the vessel gets shown, but of course with a very strange jumping. I guess that's because nobody updates the server states for that vessel anymore.
I noticed that the list was empty before I logged on. It would be interesting to see what happens if more than one client is online, will test that...
 

Majid

Active member
Joined
Oct 31, 2014
Messages
156
Reaction score
27
Points
43
Now it keeps on running when I press PER. However, if I log off and log on again with the same persister ID, my vessel is not synchronized back again. If I log on after deleting the ID file, the vessel gets shown, but of course with a very strange jumping. I guess that's because nobody updates the server states for that vessel anymore.
I noticed that the list was empty before I logged on. It would be interesting to see what happens if more than one client is online, will test that...

I have added a ShuttleA hovering a few km over BB, ISS, a DG hovering 10km over Mars.

The logout bug is a known problem that'll go away this weekend when logged off vessels are moved into an actual persister pool and then moved back into your pool.
 

Face

Well-known member
Orbiter Contributor
Addon Developer
Beta Tester
Joined
Mar 18, 2008
Messages
4,403
Reaction score
581
Points
153
Location
Vienna
I have added a ShuttleA hovering a few km over BB, ISS, a DG hovering 10km over Mars.

The logout bug is a known problem that'll go away this weekend when logged off vessels are moved into an actual persister pool and then moved back into your pool.

I managed to get two clients to show each other vessels. However, I get a 1-second jitter of about half a meter on Earth. It is pretty smooth on the moon, although still showing noticeable jitter.

On orbit, I get wild displacement every minute or so, like displayed here:
orbiter.world.2.png


However, I get that too with OMP on 2016, although with a smaller amplitude. On 2010, even the Earth orbit absolute vectors are very smooth. Perhaps something with the callback order changed accuracy of DefSetStateEx.

I'd say for small concurrent teams this is certainly a good start to try out your ideas. :thumbup:

Good luck with the persistence :cheers: .
 

Majid

Active member
Joined
Oct 31, 2014
Messages
156
Reaction score
27
Points
43
I managed to get two clients to show each other vessels. However, I get a 1-second jitter of about half a meter on Earth. It is pretty smooth on the moon, although still showing noticeable jitter.

On orbit, I get wild displacement every minute or so, like displayed here:
orbiter.world.2.png


However, I get that too with OMP on 2016, although with a smaller amplitude. On 2010, even the Earth orbit absolute vectors are very smooth. Perhaps something with the callback order changed accuracy of DefSetStateEx.

I'd say for small concurrent teams this is certainly a good start to try out your ideas. :thumbup:

Good luck with the persistence :cheers: .

I was watching your stunts. Sadly I forgot to persist my own vessel, otherwise you would have seen me going through you on on the moon. The wild displacement is by design, it only updates the states every 60 seconds for vessels in orbit, to get rid of jitter all together. But you'd have to synchronize with the other party to not burn.

The idea isn't to replicate OMP with PTP synchronized timestamps :p Someday I hope to integrate with OMP for super high accuracy "instance" like situations, but for the cost of 1 ajax call, I am going to get as much multiplayer as I can get. The next thing I am going to try is smoothing in local coordinates from the POV of the focus vessel if the distance of the spawned vessel is small.

---------- Post added at 11:57 AM ---------- Previous post was at 11:55 AM ----------

When do you think GL-01 will exit the solar system?
 

Face

Well-known member
Orbiter Contributor
Addon Developer
Beta Tester
Joined
Mar 18, 2008
Messages
4,403
Reaction score
581
Points
153
Location
Vienna
I was watching your stunts. Sadly I forgot to persist my own vessel, otherwise you would have seen me going through you on on the moon. The wild displacement is by design, it only updates the states every 60 seconds for vessels in orbit, to get rid of jitter all together. But you'd have to synchronize with the other party to not burn.

The idea isn't to replicate OMP with PTP synchronized timestamps :p Someday I hope to integrate with OMP for super high accuracy "instance" like situations, but for the cost of 1 ajax call, I am going to get as much multiplayer as I can get. The next thing I am going to try is smoothing in local coordinates from the POV of the focus vessel if the distance of the spawned vessel is small.

Yeah, replicating OMP with HTTP would be hard, anyway, because the lag gets only longer the more users and vessels you get into the single /tele list. But for the project you imagine, it should be really good enough, I'd say.

My displacement screeny shows that there is a discrepancy about what two users see during docking, if the 1-minute pings jitter so much. The one sees a nice docking, while the other sees a catastrophic collision.

Given that your concept doesn't use P2P techniques to keep lag low, why not use the "one-server-one-manipulator" concept? I.e. use the Orbiter clients to send manipulation data to the server, like keystrokes or joystick deflections, and let the actual server control the vessels, so that the overall vessel position "image" is a consistent one? You are not after real-time multiplayer, anyway, so a little control lag won't matter that much.

Otherwise I fear you'll end up having to replicate the full OMP experience to avoid those discrepancies described above.

I can only dissuade you from using relative vectors. They make things much more complicated, that's why I've deprecated them in OMP (except for attachment support). In addition, they produce the dilemma of having to mark one master vessel in a crowd, something you can't chose trivially by automatic methods.
 

Majid

Active member
Joined
Oct 31, 2014
Messages
156
Reaction score
27
Points
43
Sorry, I can't reply to your post in kind due to work but I value user experience > realism -> so I am going to treat docking as an event and transmit that to the clients so the discrepancy you describe doesn't happen (for docking). AFAIK transmitting controls -> already thought about doing this :D

---------- Post added at 01:48 PM ---------- Previous post was at 01:17 PM ----------

Also, if I am understanding you correctly: control lag for the user would be horrendous. I tried to fly a DG by using the client for visual feedback and it was awful with 1 second update intervals. It was doable with 0.25 second update interval.

---------- Post added at 01:58 PM ---------- Previous post was at 01:48 PM ----------

Given orbiter's global coordinates, is there some easy transformation I can do to display the current vessel positions in a meaningful 2D projection that I can display in a web page to display to people so people can check the server state at a glance on their phone?

---------- Post added at 02:00 PM ---------- Previous post was at 01:58 PM ----------

I guess as a start I can just map the longitude and latitude on a map for Earth.
 

Face

Well-known member
Orbiter Contributor
Addon Developer
Beta Tester
Joined
Mar 18, 2008
Messages
4,403
Reaction score
581
Points
153
Location
Vienna
Sorry, I can't reply to your post in kind due to work but I value user experience > realism -> so I am going to treat docking as an event and transmit that to the clients so the discrepancy you describe doesn't happen (for docking).

Good idea.

Also, if I am understanding you correctly: control lag for the user would be horrendous. I tried to fly a DG by using the client for visual feedback and it was awful with 1 second update intervals. It was doable with 0.25 second update interval.

What I meant was more event based. Whenever you control your vessel, gather the events for e.g. 100ms, then post them to the server object. The server (or persister) plays it back to the server-hosted object, which in turn updates its reactions. Since this only runs if somebody does something (instead of coasting or playing with MFDs), the strain on the server would not be that high.

I did a similar thing in OMP when trying to tackle the animation problem, but of course there it the round-trip was much faster. It worked pretty sweet, enabling people to control the vessels of other users. Unfortunately, it did not really fit into the P2P approach, so I deprecated that, too.
 

Majid

Active member
Joined
Oct 31, 2014
Messages
156
Reaction score
27
Points
43
What I meant was more event based. Whenever you control your vessel, gather the events for e.g. 100ms, then post them to the server object. The server (or persister) plays it back to the server-hosted object, which in turn updates its reactions. Since this only runs if somebody does something (instead of coasting or playing with MFDs), the strain on the server would not be that high.

I did a similar thing in OMP when trying to tackle the animation problem, but of course there it the round-trip was much faster. It worked pretty sweet, enabling people to control the vessels of other users. Unfortunately, it did not really fit into the P2P approach, so I deprecated that, too.

This is really interesting.. I was thinking of using GetAsyncKeyState to write a sort of keylogger (did it before) and transmit those events. The persister can also be restarted, so my initial experiments showed that I can propagate animation/(scenario file saved) state by oapiSaveScenario then transmit the blocks to the persister and have it reload the scenario... It may be worthwhile to look into the vessel hooking stuff? I couldn't find anything on OH, is this the latest on vessel hooking? : http://orbiter-forum.com/showthread.php?t=4947

In this architecture, you can have N number of persisters and each one posts its position every one second and the server responds with the vessels it should persist. If you have the serverpersister module active, you're a persister. Maybe in the future I'll release it separately with an opt-in "Allow Orbiter to persist vessels for other users" checkbox.
 
Last edited:

Majid

Active member
Joined
Oct 31, 2014
Messages
156
Reaction score
27
Points
43
New update: http://orbiter.world/orbiter.online.zip

Improved the mjd controller, decreased clock update interval to 3 seconds. Added orbital vector smoothing. Got rid of the slow update cycle, so now everything updates at 1 Hz even vessels in orbit.

It's possible now to do real-time dockings and for both parties to see real-time non-jittery updates. The plug-in handles this by treating vessels which haven't burnt in the past 30 seconds as inactive. Active vessels get rid of the jitter by simply ignoring all state changes that result in a displacement greater than twice the relative velocity. Whenever such an event occurs, the active vessel asks the inactive vessel to reconcile its position in local coordinates resulting in less jitter.

So the approaching active vessel will see no jitter, the inactive vessel will see greatly reduced jitter.
 

Face

Well-known member
Orbiter Contributor
Addon Developer
Beta Tester
Joined
Mar 18, 2008
Messages
4,403
Reaction score
581
Points
153
Location
Vienna
New update: http://orbiter.world/orbiter.online.zip

Improved the mjd controller, decreased clock update interval to 3 seconds. Added orbital vector smoothing. Got rid of the slow update cycle, so now everything updates at 1 Hz even vessels in orbit.

It's possible now to do real-time dockings and for both parties to see real-time non-jittery updates. The plug-in handles this by treating vessels which haven't burnt in the past 30 seconds as inactive. Active vessels get rid of the jitter by simply ignoring all state changes that result in a displacement greater than twice the relative velocity. Whenever such an event occurs, the active vessel asks the inactive vessel to reconcile its position in local coordinates resulting in less jitter.

So the approaching active vessel will see no jitter, the inactive vessel will see greatly reduced jitter.

Did you try that with more than 2 clients already? I guess it will cause a third observer to disagree on the relation between the other two. Especially if there is more than 1 active vessel, i.e. two approaching ISS simultaneously.

---------- Post added at 20:13 ---------- Previous post was at 20:08 ----------

:rofl: GL-01 does a funny bunny-hopping now on the second client while landed.

---------- Post added at 20:16 ---------- Previous post was at 20:13 ----------

The standard "ISS on one client, DG on another, trying to dock" jumps around in a 10 meter fashion in 1Hz now.
 

Majid

Active member
Joined
Oct 31, 2014
Messages
156
Reaction score
27
Points
43
Did you try that with more than 2 clients already? I guess it will cause a third observer to disagree on the relation between the other two. Especially if there is more than 1 active vessel, i.e. two approaching ISS simultaneously.

---------- Post added at 20:13 ---------- Previous post was at 20:08 ----------

:rofl: GL-01 does a funny bunny-hopping now on the second client while landed.

---------- Post added at 20:16 ---------- Previous post was at 20:13 ----------

The standard "ISS on one client, DG on another, trying to dock" jumps around in a 10 meter fashion in 1Hz now.

I did, because of the local shenanigans I do expect active clients to get disagreements about each other. But the "ISS on one client, DG on another trying to dock" setup should not result in 10 meter jumps o.o. If you see jumps on the ISS position, trying burning a thruster on the DG and that should smooth out the ISS position instantly.

Open one client, persist ISS.
Open another client, persist a DG. Copy the state of the DG to the ISS.

Switch over to the other client. At this point, hopefully both clients should be in sync. If you see jumps, use a thruster on the intended active client. It has a 30 second timeout, after which you become an inactive client yourself and stop applying localized smoothing to global coordinate updates (thus the 10 meter jumps).

Basically the trade off is: Given that there is agreement between two parties that one will remain inactive and one is approaching, the active party will see no jitter at all. If you see jitter, you are inactive.

The behavior for non-orbital vessels remains unchanged, can you explain the steps you took to reproduce the behavior? I created a DG on the ISS client, persisted it. Switched focus to the DG on the other client, and it remains on the ground plopping like always.

My digitalocean droplet is temperamental, I guess this isn't the intended use of their droplets :p What's your ping like to orbiter.world?
I am going to move over to Google cloud soon, so we should get much better performance from HTTP side...

But I'd love to know how you managed to get 10 m jitters on ISS in orbit. The github has the latest source, you can take a look that on local active clients if the local displacement is really high it just ignores it entirely, thus the need for reconciliation. The ISS client will see jitter, but it should be a lot less than what you'd normally see with straight up global vessel coordinates.
 
Top