Performance Issues on Python 3
Request for Comments.
TL;DR : Someone can suggest some integer math accelerating package (if existent)?
FULL HISTORY: besides never expecting bleeding edge performance, I'm somewhat concerned about the performance of the tool.
I already knew from my past experiences with Python2 that this is not made for speed, and frankly, speed is not the most important requirement for this tool. But Python3 introduced some changes that made my core business even slower - INTEGER math is almost 3 times slower than Python2 at worst case.
Changing languages is not an option for two reasons:
1) Java (or any other language running on the JRE, as JPython or Haskell, Scala or Groove) is a no go. Installing a runtime for Python is way less problematic (and controversy) than a JRE, and I have the option of using PyInstaller and save the user of doing that.
2) I don't want to introduce yet another language on the ecosystem. Hell, I was planning to use LUA to do the job (as it is used on Orbiter), but adopted Python due Blender. And since Python is also used on some other 3D packages, using anything else for this tool would be foolish.
So Python it is, and Python will be.
But my problem remains: I'm taking more than 90 minutes to LINT all my meshes on my Orbiter installation, and LINTing bigger meshes using the GUI can take more than a minute sometimes.
My first move was to give a try on numpy package - something that I dismissed at first to avoid "yet another dependency" (tm) on the project, and since the data structures are more complex than just vectors and matrices (and most of the current operations are accelerated by caching hashes), and profiling tells me that this is not (yet) the worst bottleneck (but it will be soon, as matricial math will be used later), I think I can deal with this in a second pass.
Garbage Collecting is under control - the tool is nothing more than batch jobs embedded into a GUI or TUI, there're clear points where we can free references and ask the GC to collect some memory (and yes, I profiled the solution - it's best to make micro GC at the end of a "job" than make a HUGE GC when memory became scarce - and this keeps memory usage under reasonable levels, I have scenarios where more than 1GB of RAM is needed).
So now I'm looking into you, INTEGER MATH. Profiling some mockups on Python2, Python3 and Python3+numpy I found that doing math with Python2 is far the best option. But not a viable one, as Blender and so O.T. uses Python3. =/
Using numpy.int32 and numpy.int64 appears to present the worst results, it's the slower of all options (they converted them into native Python format on the fly?).
Someone can suggest some integer math accelerating package (if existent)? My intensive use of hashes suggests me that my current bottleneck is integer math.