Apollo 11 TEI failue

[Helios]

So I've been flying the 11 mission and I've reached the point where I do the TEI burn.

I've followed the checklists to the letter and made sure that everything is where it needs to be. Yet when the burn begins, the entire CSM starts to tumble end over end.

I'm at a loss to understand what's going on here. The only thing that seems rather strange to me is that the SPS engine starts swinging wildly across all axis cause the trim gauges go nuts.

Anyone have any insight?

 

[indy91]

So I've been flying the 11 mission and I've reached the point where I do the TEI burn.

I've followed the checklists to the letter and made sure that everything is where it needs to be. Yet when the burn begins, the entire CSM starts to tumble end over end.

I'm at a loss to understand what's going on here. The only thing that seems rather strange to me is that the SPS engine starts swinging wildly across all axis cause the trim gauges go nuts.

Anyone have any insight?

Can you post your scenario before TEI? Then I can check what might be wrong with it. I really hope it's not another case of the weird erasable memory corruption some people seem to be getting, I really don't understand that at all. But maybe (and hopefully) it's just the DAP not properly set up.

 

[Helios]

Sure, here's a quicksave I took two minutes before TEI. I had to paste the scenario file in Google Docs due to character limits here.

https://docs.google.com/document/d/1UJtoz-wSIVKRRtpOWyDJGi03L-iT2YkXNJ2BhPbwegc/edit?usp=sharing

 

[indy91]

As I feared, something has messed up very specific erasable memory locations of your CMC. In this case numbers that usually stay constant throughout the flight and are vitally important for the TVC DAP to work. I've seen that in several scenarios in the last time, never in my own though. And the Virtual AGC hasn't been changed in the last 10 months and not for several months before that either. So it's really weird.

Anyway, I've fixed the scenario and attached it to this post. No guarantee that other parts of the erasable memory aren't also broken, but this is what I have found by looking at the right place and TEI seems to be working fine now.

EDIT: Did anything unusual happen since the last time the TVC DAP would have been used (likely LOI-2)? Like, did you get a program alarm or restart light in the CMC?

 

[Helios]

Now that you mention it, I did get some weird program alarms while setting up for the burn. When doing Star Select I got a PGNS flag and the Sextant was pointing nowhere near the target stars. I had to do a P51 realign to set it up properly again.

Anyway, scenario works and Columbia's on her way home now. Thanks a lot!

---------- Post added at 03:19 PM ---------- Previous post was at 01:51 PM ----------

EDIT: Did anything unusual happen since the last time the TVC DAP would have been used (likely LOI-2)? Like, did you get a program alarm or restart light in the CMC?

Okay I'm in the first 10 hour rest period now post TEI and the DKSY did display both a program alarm and a restart light. Not sure what that means but it did happen.

 

[abr35]

Is there any connection between your restart lights and either time acceleration or rapidly switch panels? I have had the same issues as you. Varies from restart lt, to restart lt and program alarm, and sometime P00 even terminates.

I forget the alarm code I would get but I believe it was a waitlist overflow. I could force the alarm and post it.

 

[Helios]

Not that I know of, but I wouldn't be surprised if time acceleration has something to do with it.

---------- Post added at 06:37 PM ---------- Previous post was at 05:06 PM ----------

Alright after further testing, it seems the flags come up during 10x time acceleration. Program 00 terminates and a V37 N00 results in P00 running with a solid COMP ACTIVITY light but then terminating with a PROG alarm light. The PNGS light on the SXT comes on as well.

SV update seems to fix the problem as well as doing a V69 to restart the CMC, though the Flashing 37 remains.

 

[indy91]

I had forgotten to merge two Virtual AGC updates into the NASSP version of the Virtual AGC (we use a slightly modified Virtual AGC, so I can't just copy the files from them). There is a small chance that it will fix the issues.

Different question: Do you two have the "Multi-threading in time acceleration" option checked in the NASSP config menu? I am not sure if that is better or worse or anything, just trying to figure out if we have any config differences, because I certainly have never had these issues with the erasable memory and program alarms before.

 

[Thymo]

I do sometimes notice restarts when coming in and out of time acceleration.
I have not been able to consistently reproduce this issue but my best guess it has something to do with the parity ropes that were added some time ago.

 

[abr35]

I had forgotten to merge two Virtual AGC updates into the NASSP version of the Virtual AGC (we use a slightly modified Virtual AGC, so I can't just copy the files from them). There is a small chance that it will fix the issues.

Different question: Do you two have the "Multi-threading in time acceleration" option checked in the NASSP config menu? I am not sure if that is better or worse or anything, just trying to figure out if we have any config differences, because I certainly have never had these issues with the erasable memory and program alarms before.

I do have multi-threading enabled. The alarms are at their most frequent during LM Activation through to landing - when the most computers are running simultaneously.

 

[kerlix]

Do you notice any reduction in framerate once both computers were running? Essentially simulating what happens during a timewarp?

Or is it just with both computers AND timewarp occurring simultaneously?

I admit, I know very little about this but had been an avid user of NASSP long ago, before all the great updates provided by Indy and others. I was following this thread and I thought of the above two scenarios as possible culprits. Then again, it could amount to nothing but I figured I'd ask to at least attempt to distill down the problem and maybe find the issue or the road to the issue.

May just help me out in the future when I get a rig back that can handle NASSP like a champ.

Sent from my Moto E (4) using Tapatalk

 

[Helios]

Well I'm not sure what happened but I've had zero problems with the AGC from the MCC-5 burn on. Save for an occasional PNGS flag (though that's probably because I somehow ended up doing a PTC at near gimbal lock)

Now it's time to go home.

---------- Post added at 11:05 PM ---------- Previous post was at 09:57 PM ----------

Okay not home just yet. The CM RCS refuse to fire. While I could just bite the bullet and go ballistic, I'd rather not cook the crew, lol.

I'm like 90% sure I missed a checklist step somewhere but for the life of me I can't figure out what. Any ideas?

---------- Post added 11-16-18 at 01:31 AM ---------- Previous post was 11-15-18 at 11:05 PM ----------

And found it. Didn't enable the RCS logic cb's. Incidentally, it seems the CSM Sep is missing a couple items, like enabling the aforementioned cb's as well as SEQ EVENT CONT ARM 1 and 2 cb's (the latter of which need to be enabled for SM jettison apparently)

 

[indy91]

Ryan (rcflyinghokie) was working on Apollo 11 checklists, when he got really busy IRL. I think he got up to the lunar landing or shortly before that. So up to that point the checklists are probably in a really good state, everything after that will have more errors and will be a bit outdated w.r.t. new features in NASSP.

And an unstable framerate might have to do with the issue, which would explain why I haven't ever experienced it. Apart from the CSM main panel I always get a stable 60fps. The CSM main panel and NASSP in general are very CPU heavy, which hopefully can be improved at some point soon. Orbiter 2016 itself is probably more GPU heavy. But yeah, the Virtual AGC (or the way we run it in NASSP) might not like unstable framerates in addition to switching in/out of time acceleration. What exactly would be causing the specific behavior with restarts and program alarms I don't know.

 

[GLS]

But yeah, the Virtual AGC (or the way we run it in NASSP) might not like unstable framerates

If it is doing integrations/filters/etc, then the frame rate will affect the results. I've been working on handling a "non-fixed" frame rate in SSU recently and by calculating "in real-time" the coefficients of the discrete-time transfer function (because the sampling rate varies with frame rate), I'm getting enough stability to fly the reentry at 10x. I made a class to implement a 1º order digital filter, but it still needs some work, especially in initializing the filter... :uhh:
If the frame rate is indeed the cause of your issue, we can have Marty McFly send the filter from the 1970s to the 1960s.

 

[indy91]

If it is doing integrations/filters/etc, then the frame rate will affect the results. I've been working on handling a "non-fixed" frame rate in SSU recently and by calculating "in real-time" the coefficients of the discrete-time transfer function (because the sampling rate varies with frame rate), I'm getting enough stability to fly the reentry at 10x. I made a class to implement a 1º order digital filter, but it still needs some work, especially in initializing the filter... :uhh:
If the frame rate is indeed the cause of your issue, we can have Marty McFly send the filter from the 1970s to the 1960s.

Hmm no, I don't think it is that. Any integrations and digital filters the AGC does/uses are only used during powered flight, but all these issues happened during coasting flight and time acceleration (changes). And it's fixed memory AGC software anyway, nothing we could change about filters/integration schemes. All we do is feed the AGC with accelerometer data etc. during powered flight, rest is done by the unmodified, original AGC software.

We are using multithreading for the Virtual AGC cycles, if we don't then time acceleration would be nearly impossible. After all the Virtual AGC timestep function has to be called about 85333 times per second (11.7 microseconds machine cycle). Maybe there is a cycle dropping or sync issue that has always been there with multithreading. If this is so, then some more recent-ish Virtual AGC features like TC Trap (check guard against infinite loops) could cause restarts and program alarms that didn't happen before this stuff was implemented.

@Helios and @abr35 and others who had the same issue, it would be great if you could update to the most recent NASSP 8.0 Alpha release. This now has the two Virtual AGC updates I had forgotten to implement in our Virtual AGC files. One of these has some additional alarm data being copied to an erasable memory location. So if the issue happens again in a scenario with the updated Virtual AGC then it will be easier for me and thewonderidiot (from the Virtual AGC project) to debug and maybe find the source of the problem.

 

[abr35]

Flying 11 again on the most recent NASSP build and still having restart issues. Program alarm 1107 Phase Table Failure has come up with the restart and my DAP code was corrupted this time - fortunately an easy fix.

Edit: If anyone is still following this bug - disabling the multi-threading in time accel solved the issue. The AGC restarts were caused by multi-threading.