“Where’s the .exe?” – Why NAM 37 Ditched It For the .jar

The NAM Team, after almost 3 years of what is affectionately known in the software and media world as “development hell“, finally released NAM Version 37 earlier this month.  This new release brought a number of significant changes, including a new installer and a new file architecture.  In this regard, it’s probably the most dramatic change the NAM has seen since the installer was first introduced with NAM 19, almost 15 years ago (September 2005!), and the infamous NAM 31 release (AKA “NAM Vista”) from March 2013.  Unlike NAM 31, however, aside from some user confusion regarding some of the changes, the new version itself has proven astonishingly solid–solid enough that the usual cadre of users who download the NAM “just for the bugs” (much like those who watch NASCAR races “just for the wrecks”) are bored out of their skulls.

The two main sources of user confusion are, of course, the new installer itself, and the requirement to patch the game with a Large Address Aware (LAA) patch, also known as a “4GB Patch”.  This post is primarily designed to address the former, though in doing so, must inevitably cover the latter.

Indeed, one of the questions we’ve encountered regarding the installer has been a simple, almost existential one: “where’s the .exe?”

The answer to that question is that there is no .exe installer for NAM 37.  Instead, since we are now using a Java-based installer, the NAM 37 installer is a .jar file.

This answer, of course, has provoked more questions, and there are segments of our userbase who are probably thinking, “why abandon something that’s worked for the NAM for so many years?”  The truth of the matter, however, is that our .exe installer hasn’t “worked for us” for some time now, and recent releases and our experiences during NAM 37’s protracted development cycle effectively required that we jettison it.

Since the introduction of installers to the NAM ecosystem with NAM 19, courtesy of long-time community Renaissance man Andreas Roth, the NAM Team has religiously used the open-source Nullsoft Scriptable Install System–better known simply as NSIS–whose end product is the familiar Windows executable (.exe) file.  NSIS provided us a more robust system than the infamous Clickteam Installers that are the bane of many modern-day SC4 players installing custom content, in addition to providing extremely efficient LZMA compression–the same algorithm that powers 7-zip’s .7z format.  With the move back to the “Monolithic NAM” paradigm (albeit in a greatly expanded form) with NAM 31, our NSIS scripts were given a massive overhaul by z, the architect of the current version of the NAM’s Traffic Simulator Plugins.  That overhaul incorporated a number of highly advanced NSIS features–including quite a few that aren’t even documented by the system’s developers, to allow for things like checking the user’s game version, or if they had certain transit stations installed in their Plugins folder.  (And if you’re curious to see it, here is the NAM 36 version of it–it’s almost 7200 lines long.)

When z had to retire from NAM development, during the NAM 33 cycle, the task of maintaining and updating that highly advanced script fell to those of us still left on the development team–which contracted considerably during that marathon cycle.  And while there were a handful of us who had worked with NSIS before, there were so many moving parts and complexities in the “Monolithic NAM” script–and with that formulation of the Monolithic NAM itself–that, over time, with each addition, things became more and more broken.  The issue of crosslinks between various either/or options (i.e. Maxis Rail vs. RealRailway, Maxis Highway vs. Maxis Highway Override, Monorail vs. Bullet Train, Default El-Rail vs. El-Rail Alternate Implementation, etc.), as well as the dizzying array of texture options in the mod at that time, were already a storm in the making, and they quickly escalated.

With the long-lived NAM 36 release, we routinely began to encounter tech support cases in which users somehow ended up missing critical core NAM files after installation, including the Individual Network RULs file and the NAM Controller–two of the most critical files in the mod, which effectively allow the placement of all the NAM’s additions to the game’s transportation networks.  The installer was seemingly developing a mind of its own, and surreptitiously skipping these files during the installation process at least some of the time.  It had effectively become a ticking time bomb in terms of when it would break entirely, and it became obvious we needed to replace it.

On top of all this was the matter of support for users running non-Windows OSes, particularly the small but very passionate contingent running macOS/OS X.  Long-time readers of SimTarkus know the checkered history of the NAM and macOS in recent years.  From NAM 19 up through NAM 30, the NAM had separate “Windows” and “Mac” versions–the former used the installer, while the latter was simply a .zip for manual installation.  The massive increase of structural complexity with NAM 31, however, meant that manual install was basically impossible (even for members of the dev team!), and our ability to provide a suitable installation experience for the Mac userbase between NAM 31 and NAM 36 depended on (a) no weird CTD-causing file encoding issues caused by Apple or Aspyr changes to the OS or the game, (b) having a team member or sufficiently technical community member with access to a Mac (memo for NAM 32, no one for NAM 33, Mero90 for NAM 34, and me for NAM 35 and 36), and perhaps most critically, (c) our reliance on Wineskin (the “Mac” version was literally the Windows installer wrapped in a Wineskin layer) and its compatibility with the latest versions of macOS/OS X.  That latter point became a serious problem beginning with macOS High Sierra, and became completely unreliable for Mojave–Catalina’s even more massive changes (which SC4 itself unplayable for a time) were insurmountable.

With NAM 37’s development being seemingly interminable, we initially planned to limp our NSIS installer along for one more release, and replace it for NAM 38.  The first build of NAM 37–Alpha Build 01 (“a01” in NAM speak), dating back to March 31, 2019–used the NSIS installer.  Unfortunately, the changes between NAM 36 and this initial stage of NAM 37 ended up proving fatal–the NAM 37 a01 installer absolutely refused to install the NAM Controller and other key files almost 100% of the time, and the complexity of the script made the prospect of fixing this issue virtually impossible.

We did, however, have a ready-made solution in hand–a Java-based installer that long-time NAM and BSC member Daeley (of Daeley’s Advanced Menu Navigator (DAMN) fame) had worked up way back in 2010, based off his work for the former SC4 Devotion LEX DVD.  While this system couldn’t support anything as complex as our old NSIS installer, it had the distinct advantage of being completely scriptless, instead using character combination flags in front of directory names to handle the various options.  This meant that with the right file architecture, the release engineering process would be effectively painless, and could be handled by virtually anyone on the team, if need be.  Additionally, it also meant that we could use a single installer for all OSes, finally allowing us to handle macOS without issue.

There were a few drawbacks we knew we would encounter with this system–two particularly big.  The first, effectively minor one, meant that with all the “either/or” options that had plagued the NSIS-based Monolithic NAM, we would effectively need to whittle those down to just one, jettisoning the others from the main part of the mod itself.  We conducted a poll at Simtropolis in April 2019, which, along with an earlier such poll from 2013, determined that the “Maxis Highway or Maxis Highway Override” either/or option would be the only left standing in NAM 37, being that our userbase was nearly in a 50/50 split between the two options.

Here are the 2019 poll results (146 responses):

Additional/Custom Bridges 117 87.97%
Diagonal Streets 117 87.97%
Street Addon Mod (SAM) 115 86.47%
Network Widening Mod (NWM) 114 85.71%
Rail Viaducts 112 84.21%
PedMalls 111 83.46%
Flexible Underpasses (FLUPs) 109 81.95%
Elevated Road Networks 106 79.70%
RealHighway Mod (RHW) 106 79.70%
Wide-Radius Curves (WRC) and Multi-Radius Curves (MRC) 106 79.70%
One-Way Road Arrow Mods 105 78.95%
RealRailway (RRW) 105 78.95%
Ground Light Rail (GLR)/Tram 103 77.44%
FLEX Turn Lanes (FTL) and QuickTurn 99 74.44%
Ground Light Rail/Tram Dual Networking (i.e. GLR-in-Avenue, etc.) 99 74.44%
Fractional Angled Networking (FAN) 96 72.18%
Turn Lane Extension Pieces (TuLEPs) 96 72.18%
Elevated Rail-over-Road/Avenue Dual Networking 87 65.41%
Single-Track Rail (STR) [RealRailway Version] 86 64.66%
Maxis Highway Override (Project Symphony) 70 52.63%
Underground Rail (URail) 63 47.37%
High Elevated El-Rail and High Elevated Monorail/BTM 62 46.62%
Elevated Rail Alternate Implementation 60 45.11%
Rural Roads Plugin (RuRP) 60 45.11%
High Speed Rail Project (HSRP) 55 41.35%
Auto/Semi-Auto Turn Lanes (the old ones) 47 35.34%
Euro Textures 47 35.34%
Maxis Highway (Standard) Interchanges and Ramps 45 33.83%
Alternate Road Texture Sets (beyond standard US or Euro) 42 31.58%
Canal Addon Mod (CAN-AM) 41 30.83%
Bullet Train Mod (BTM) 36 27.07%
Maxis Rail Features (Legacy/Non-RRW) 32 24.06%
Single-Track Rail (STR) [Maxis Rail Version] 16 12.03%
None of the above 0 0.00%

 

And the 2013 poll results (459 responses):

Roundabouts and Other Non-Highway Interchanges (87.9%)
Diagonal Streets (85.7%)
Wide Radius Curves (85.1%)
Network Widening Mod (NWM) (83.8%)
Street Addon Mod (83.3%)
Real Highway Mod (RHW) (82.0%)
NAM Custom Bridges (81.1%)
Turning Lanes (80.5%)
Flexible Underpasses (FLUPs) (76.3%)
One-Way Arrow Mods (75.0%)
NAM Transit Stations (73.5%)
Elevated Road Networks (70.4%)
Fractionally Angled Networks (FANs) (70.4%)
Ground Light Rail / Tram (69.7%)
TuLEPS (69.5%)
Ped Malls (67.3%)
Custom Road Textures and Markings (65.4%)
Real Highway Regional Transport View (62.9%)
Viaduct Rail (61.8%)
Maxis Highway Interchange & Ramp Additions (53.3%)
Single Track Rail (STR) (52.6%)
Extended Station Queries (the large NAM style) (51.5%)
El Rail over Road Networks (50.4%)
Maxis Highway Override (Project Symphony) (49.8%)
Rural Roads Plugin (47.1%)
High Speed Rail Project (HSRP) (42.1%)
El Rail Alternate Implementation (37.3%)
Underground Rail (35.5%)
Bullet Train Mod (BTM) (33.1%)
Double-Height Elevated Rail (31.8%)
Canal Addon Mod (CAN-AM) (25.7%)
Double-Height Monorail / Bullet Train Mod (25.2%)

The matter of the new installer being Java-based and requiring the Java Runtime Environment (JRE) was something we expected might come as a slight shock (side note–the best versions of the JRE out there are AdoptOpenJDK’s builds), but the simple fact of the matter is that, ever since the Controller Compiler debuted in NAM 32 (January 2014), where it was run by our NSIS installer, Java has been an integral part of the NAM installation process.  The Controller Compiler–as well as the Traffic Simulator Configuration Tool (TSCT)–are Java-based utilities.

The bigger downsides, however, were the fact that the new installer did not have the ability to call the Controller Compiler, which meant we had to install a full-blast NAM Controller by default.  And with the various upgrades to adjacency stability, especially on the RHW end, the NAM Controller now ran 281MB–a size which requires the user use the 4GB Patch, or risk CTDs (unless they wish to manually run the Controller Compiler and whittle down their options).  And we also could not automatically call the 4GB Patch, either.  Our hope was that, since the last few versions of the NAM (including NAM 36) had prompted users to install the 4GB Patch if they didn’t already have it applied, that they would already have it in place when they installed NAM 37.  Unfortunately, that patch option included an “opt-out”, and it appears a number of users either chose to opt out, or have reinstalled/changed to a different version of the game (i.e. going from CD to GOG or Steam) without patching their new installation, so we’ve encountered a few of these cases–though those have usually been easily rectified.

Speaking as an architect of this whole process, overall, I’m actually pleasantly surprised by how smooth the rollout of NAM 37 has been–especially having lived through the NAM 31 debacle.  And as both the development team and the users become more familiar with this system, we can better document the answers to the questions that come up, and it permits more frequent releases (while we don’t publicize release dates, it’s very safe to say that NAM 38 will not take another 3 years, especially as we’ve gone to an Agile-style development approach post-NAM 37), I suspect some of the initial wariness toward these changes will wane, and the .jar will be as accepted as our old .exe was.

-Tarkus

NAM 37 Now Available

After almost 3 years, we’ve finally done it.  NAM 37’s official release has arrived at all three of our official distribution points: SC4 Devotion, Simtropolis, and ModDB.  You can read all about it here.  On behalf of the whole NAM Team, we hope you enjoy this new release!

-Tarkus