[ubuntu-mono] [Bug 412503] Re: OpenBve+Class 323 does not support plugin DLLs on Linux

[Aidan Fell]

As far as I am aware, there is no current available 'working' workaround
for this? Despite the .so files and the Class 323 config file being
available?

-- 
You received this bug notification because you are a member of Ubuntu
CLI/Mono Uploaders, which is subscribed to openbve in Ubuntu.
https://bugs.launchpad.net/bugs/412503

Title:
  OpenBve+Class 323 does not support plugin DLLs on Linux

Status in bve-train-br-class-323 package in Ubuntu:
  Opinion
Status in openbve package in Ubuntu:
  Opinion

Bug description:
  Binary package hint: openbve

  Routes and trains distributed for the BVE Trainsim (proprietary) and
  OpenBve (in Ubuntu) railway simulators consist primarily of text-based
  configuration and model files, which in turn reference to PNG/BMP/JPG
  images and WAV/OGG sound files.

  In addition, the BVE Trainsim version 4 format allows each
  simulation/scenario to be programmatically extended by a compiled
  plugin.  The format defines that this plugin is compiled to a
  Win32/PE+COFF/i386 DLL file.  OpenBve (when also running on Win32)
  attempts to be backwards-compatible and load these Win32-based DLL
  plugins.  Each DLL plugin exports a series of defined callbacks (the
  BVE4 plugin API) that can override the internal (default) BVE4/OpenBve
  state engine;  for example to simulate country-specifc, or train-
  specific safety interlocks and cab-based signalling systems.  The
  naming of plugins reflects the Japanese Automatic Train Stop (ATS)
  signalling system, where BVE4 originated from.

  Inputs to the plugin arrive every frame, giving the state of the
  train, and ".Beacon" updates (one-way transponder messages placed
  along the track) , and from keypresses submitted by the user (the
  keys: Insert/Delete/Home/End/PageUp/PageDown/Spacebar and 2 to 9 are
  exposed).  Outputs are in the form of forced changes to the state
  machine (for example, hitting the brakes when passing a red signal),
  updates to the list of sounds being played, and the exporting of up to
  256 variables in the format ats0-ats255 to other parts of the
  simulation.  The variables can be referenced wthin a train's cab-panel
  description or internal/external 3D model to reflect the operational
  state of the train;  for example showing a raised, or lowered
  paragraph, or by drawing individual raindrops on the windscreen.

  These plugins are generally non-free, non-distributable, and it is
  very unusual that the source-code is available.  In the case of the
  'OS_Ats1' plugin (last release 1.3 in 2005), source code *is*
  available and I have had communications with the author of the plugin
  to investigate releasing the source code under a DFSG-free licence.
  Possibly the GPL (already confirmed) or ideally placing the plugin
  code in the Public Domain (tentatively confirmed).  The 'bve-train-br-
  class-323' train already in Ubuntu would normally be shipped wth a
  binary plugin called 'UKMUt.dll' to control its extended
  functionality.  The 'UKMUt' (non-free) and 'OS_Ats1' (soon to be free)
  plugins share about 90% equivalent capabilities and I have written a
  configuration file that allows 'OS_Ats1' to be a close-enough work-
  alike for the missing 'UKMUt' plugin. Branches of OS_Ats1 are packaged
  up in Git at:

    http://github.com/sladen/bve-plugin-os-ats/tree/debian

  When a train developer chooses to request the additional services
  provided by a plugin, a file named 'ats.cfg' is included in the
  appropriate 'Train/' directory (eg.
  '/usr/share/games/bve/Train/BR_Class_323/ats.cfg') and this file
  contains a single undelineated line giving the name of the plugin
  including its ".dll" suffix ('OS_Ats1.dll').

  The code path used by OpenBve to load and link to these plugins is to
  first extract the (optional) plugin name from the 'ats.cfg' file, then
  to thunk from the portable C#/CLR code to and from the Win32 DLL via
  an intermediate proxy:

    'OpenBve.exe' (CLR/CLI) <> 'AtsPluginProxy.dll' (Win32/i386) <>
  'Something_Ats.dll' (Win32/i386)

  The upstream binary distribution includes a non-portable binary
  compilation of 'AtsPluginProxy.dll', this step is only present as it
  appears that CLR effectively resolves DllImports at *compile time*,
  rather than allowing *dynamic importing* (such as dlopen()).
  Upstream did not really envision plugins being used on platforms other
  than Win32+i386 and so the codebase has included long-term checks and
  *hard* warning messages that active loading from being attempted:

    if(Program.CurrentPlatform != Program.Platform.Windows)
      "...plugins are not supported on operating systems other than Windows."
    if (IntPtr.Size != 4)
      "...plugins are not supported in 64-bit environments."

  I can confirm that refining these two checks allows the OpenBve code
  base to load and use a natively compiled version of 'OS_Ats1.so', via
  (an also natively-compiled) 'AtsPluginProxy.so'---basically
  unmodified.  The patches to narrow these two checks along with the
  other tweaks were provisionally presented to upstream for review (see
  the following Git branch):

    http://github.com/sladen/openbve/tree/native-plugin-support
    http://www.paul.sladen.org/openbve/patches/1.2.0.0-native-plugin-support/

  Some of the patches were accepted and integrated.  However, in the
  case of specifically removing/refining/fixing the Win32 check to allow
  loading of _natively compiled_ .so plugins on Linux, I have been
  informed that this would constitute a fork, requiring the renaming of
  .deb packages and removal of all references to OpenBve from the
  packages.  The renaming/debranding away from the upstream OpenBve is
  not desirable, even if it has been done in the cases of
  Iceweasel/abrowser as a last resort---additionally upstream have
  indicated that they would likely cut off future communication if the
  fork route was chosen.

  Upstream have indicated they wish to see plugins continue to be
  distributed *only* as Win32/PE+COFFs DLLs.  The argument, as I
  understand it, is that the selection of the format for images, routes
  files and plugins was chosen by BVE Trainsim 4 (a proprietary program)
  and that those chosen (binary/text) formats are supported only for
  backwards compatibility reasons.

  A fear was presented that allowing other compiled formats of plugins
  (eg. Libc/ELF/armle) might lead to additional incompatibility issues;
  an MS Windows user might complain that a plugin compiled for some
  other processor+ABI type and called MyPlugin.so would not load (as a
  binary) on /their/ system.  A newer release of OpenBve now includes a
  minimal check for an 'MZ' (DOS executable) header.

  Technically, the solution for "making [source available] plugins work
  on Linux" is very easy (removal of the overly-zealous Platform check),
  however, politcally it is a very different issue.  In essence, whether
  to achieve backwards-compatibility for plugins at the compiled binary
  level, or at the source code level is a difference of opinion.
  Upstream have provided a list of ten (non-free) binary i386+PE+Win32
  DLLs that must be proven working for any proposed solution to be
  deemed acceptable.

  In order to satisfy OpenBve's upstream's requirements and those of
  Debian Policy, it will been necessary to cross-compile any plugin
  source-code to an i386+PE DLL (build dependency on 'mingw32').  Then
  to load this foreign i386+PE DLL shared library it will be necessary
  to parse the PE headers.  For platforms !i386 it will be necessary to
  use emulation.

  Load leaf-node DLLs (library which do not then go on to import any
  further external references) is relatively easy;  the classic example
  is the 'win32-codecs' package and the loaders in FFMPEG.  However,
  upon analysis most of the reviewed plugins further import Win32 API
  functions;  such as to read and locate their own configuration files.
  Implementations of these further imported functions must be provided
  and the plugin "format" provides no restriction on what can be used.

  Long-term it appears possible to write (from scratch) aPublic Domain
  pure C# PE parser, a pure C# i386 user-space interpreter (x86
  emulator) and pure C# stub implementations of any imported Win32
  functions, as these are discovered.  In addition, bundling a built-in
  emulator+loader would on Linux/* but also Win64/amd64, Win64/ia64 and
  WinCE/* and Mac OSX without additional dependencies;  not of these
  platforms have plugin functionality either.  This avenue of
  development will be extremely time-intensive and although a very small
  amount of work has already been done in this direction, it boils down
  to reimplementing a subset of Qemu and Wine in a new language (C#).

  A shorter-term (quicker) option would be to just *use* Wine to parse
  and load plugins and then to (crucially) resolve any Win32 imports---
  as Wine has a fairly complete set of of Win32 API function
  implementations already.  On !i386, Qemu can then be used in addition,
  to provide the dynamic binary translation from i386 to the native
  instruction set.  It should be possible to write a loader Win32 .exe
  program (somewhat equivalent to 'AtsPluginProxy'); this could be
  started inside Wine and to communicate and receive requests over a TCP
  connection opened by the parent CLR/CLI OpenBve.exe instance.  This
  would ensure that cross-binary and cross-instruction set data
  exchanges only happen using a clearly defined wire, rather than via
  the exchange of pointers to unmanaged in-memory arrays.

  Such a solution would have to 'Depends: wine' an entail introducing a
  large download for those demanding plugin support (just to support a
  10 kB plugin).  It would however work on Win64/amd64 systems without
  Wine, where the equivalent backwards-compatible Win32 API is already
  provided, and/or emulated.

  In summary; the availability of out-of-the-box BVE plugins on Linux is
  not a technical problem, but a legal/political one: (a) source code of
  plugins is not generally available, nor DFSG-free for packaging, and
  (b) OpenBve actively restricts plugins to only one specific binary
  format and CPU instruction set.

  We have a partial solution for (a) in the form of the source code and
  support of the OS_Ats1 author;  but for (b) more work still needs to
  be done to support the requirements of upstream.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/bve-train-br-class-323/+bug/412503/+subscriptions