Bug#800622: game-data-packager: Quake 2 & 3 (GOG), make-template output

[Alexandre Detiste]

Package: game-data-packager
Version: 42
Severity: wishlist
Hello g-d-p devs!
Quake 2 & 3 just got added to GOG catalog, and I couldn’t resist the urge to buy them instantly ;)

Thanks !

I see there is here baseq2, xatrix & rogue that are already supported;
but also action, arena, capture, chaos, eraser, jail, kick, pb2 & rover.


Simon:
you're right about the fact that the .dll & .exe should
had been listed in make-template output, even as "disabled";
now I have no idea if these mods require a game.so or not;
were the presence of a .dll would had told me.


What's really weird is that some game.so leaked in GOG's windows archive:
- arena/gameaxp.so
- arena/gamei386.so
- arena/gamemips.so
- arena/gamesparc.so
- rover/gamei386.so

But we can't use some random object code without any source.

~~~

There's also music/track02.ogg till music/track21.ogg
where quake2-music expect
/usr/share/games/quake2/baseq2/music/02.ogg
/usr/share/games/quake2/baseq2/music/03.ogg
/usr/share/games/quake2/baseq2/music/04.ogg
/usr/share/games/quake2/baseq2/music/05.ogg
/usr/share/games/quake2/baseq2/music/06.ogg
/usr/share/games/quake2/baseq2/music/07.ogg
/usr/share/games/quake2/baseq2/music/08.ogg
/usr/share/games/quake2/baseq2/music/09.ogg
/usr/share/games/quake2/baseq2/music/10.ogg
/usr/share/games/quake2/baseq2/music/11.ogg

The extra files would map somehow to some tracks of these two CD's;
(& some track are re-used on several CD too):

http://musicbrainz.org/release/a1eb0af4-6348-4aa6-a586-4f41a33c5dea
http://musicbrainz.org/release/7590e38c-f203-425a-b726-37a55b6d2937


Alexandre Detiste

[Simon McVittie]

Post by Alexandre Detiste
but also action, arena, capture, chaos, eraser, jail, kick, pb2 & rover.

There isn't necessarily a great deal of point in packaging these: they
likely won't work without game-code, which for Quake II means native code.

Post by Alexandre Detiste
now I have no idea if these mods require a game.so or not;
were the presence of a .dll would had told me.

Anything that "changes the rules" (so, any mod with new weapons,
monsters etc.) will need a game.so.

S

[vv221]

Post by Alexandre Detiste
There's also music/track02.ogg till music/track21.ogg
where quake2-music expect
/usr/share/games/quake2/baseq2/music/02.ogg
/usr/share/games/quake2/baseq2/music/03.ogg
/usr/share/games/quake2/baseq2/music/04.ogg
/usr/share/games/quake2/baseq2/music/05.ogg
/usr/share/games/quake2/baseq2/music/06.ogg
/usr/share/games/quake2/baseq2/music/07.ogg
/usr/share/games/quake2/baseq2/music/08.ogg
/usr/share/games/quake2/baseq2/music/09.ogg
/usr/share/games/quake2/baseq2/music/10.ogg
/usr/share/games/quake2/baseq2/music/11.ogg
The extra files would map somehow to some tracks of these two CD's;
http://musicbrainz.org/release/a1eb0af4-6348-4aa6-a586-4f41a33c5dea
http://musicbrainz.org/release/7590e38c-f203-425a-b726-37a55b6d2937

I listened to Ground Zero and The Reckoning soundtracks online, and it
appears that track12.ogg to track21.ogg map to the Ground Zero
soundtrack as described here:
https://musicbrainz.org/release/7590e38c-f203-425a-b726-37a55b6d2937

[Alexandre Detiste]

control: clone -1 -2
control: retitle -1 support Quake2 GOG package
control: retitle -2 support Quake3 GOG package

This is a lot of work and Q2 support need an extra trick for the .ogg files,
so splitting this bug in two.

[Alexandre Detiste]

control: tag -1 pending

Hi,

game-data-packager should now support the GOG release.

Can you test ?

lgogdownloader integration should also work



The HTML documentation is now also included in the package
some files have 'Windows' & 'Linux' versions;
the only difference being these two tags alternativelly stripped out:

<meta name="generator" content="Microsoft FrontPage 4.0">
<meta name="generator" content="Adobe GoLive 4">


There's also specific html pages versions for "Quake3 Team Arena"
but those are currently ignored.



There's also this ...
167 e2fd03b57cd200b571849f7ac0f5178c baseq3/q3key
does that means that all GOG users share the same "unique" serial ?

ioquake3 doesn't seems to care at all anymore about the key anyway.

Greets,

Alexandre

[vv221]

On Tue, 13 Oct 2015 22:43:58 +0200 Alexandre Detiste

Post by Alexandre Detiste
game-data-packager should now support the GOG release.
Can you test ?

There seems to be a problem with the GOG version support by git version
of g-d-p. It extracts the game data, identifies two pak0.pk3 files, but
then loops back: archive extraction again, .pk3 files identification, etc.

I put a copy of my console returns in the attached g-d-p_quake3.log
file. I had to interrupt the process via Ctrl+C, or I guess it would
have kept looping indefinitely.

Post by Alexandre Detiste
There's also this ...
167 e2fd03b57cd200b571849f7ac0f5178c baseq3/q3key
does that means that all GOG users share the same "unique" serial ?

That’s right as far as I know.
I guess any game owner can request a really unique serial from GOG.

[Simon McVittie]

Post by vv221
There seems to be a problem with the GOG version support by git version
of g-d-p. It extracts the game data, identifies two pak0.pk3 files, but
then loops back: archive extraction again, .pk3 files identification, etc.
I put a copy of my console returns in the attached g-d-p_quake3.log
file. I had to interrupt the process via Ctrl+C, or I guess it would
have kept looping indefinitely.

Weird. Could you please try the same thing with DEBUG=1 in your
environment, for instance via a command-line like this?

DEBUG=1 game-data-packager quake3 ...

Thanks,
S

[vv221]

Post by Simon McVittie
Weird. Could you please try the same thing with DEBUG=1 in your
environment, for instance via a command-line like this?
DEBUG=1 game-data-packager quake3 ...

You can find the result of a debug run in the attached .log.gz file.

I interrupted the process manually, not knowing if and when it will end,
but I’m running it again this time letting it go as long as it "wants",
just in case I’m missing something.

[Simon McVittie]

Post by vv221
You can find the result of a debug run in the attached .log.gz file.

Thanks. It looks as though there may be two bugs here:

* a bug in the metadata: we ask innoextract to extract files from
setup_quake3_2.0.0.2.exe, but then we do not actually find all the
files that our metadata says it should provide, possibly because the
names seen in the archive (extras/help/manual/htmlpages/*) are not a
case-insensitive match for the names we ask innoextract to unpack
(Help/Manual/htmlpages/*);

* a bug in the g-d-p code: after that has happened, the g-d-p code does
not recover gracefully, but instead tries to extract the same files
from the same archive in order to provide some other file that might
not actually be found. We should extract each archive at most once.

S

[Alexandre Detiste]

Post by Simon McVittie

Post by vv221
You can find the result of a debug run in the attached .log.gz file.

* a bug in the metadata: we ask innoextract to extract files from
setup_quake3_2.0.0.2.exe, but then we do not actually find all the
files that our metadata says it should provide, possibly because the
names seen in the archive (extras/help/manual/htmlpages/*) are not a
case-insensitive match for the names we ask innoextract to unpack
(Help/Manual/htmlpages/*);

Hi,

This commit, while fixing something else, likely broke it:

http://anonscm.debian.org/cgit/pkg-games/game-data-packager.git/commit/data/quake3.yaml?id=13c256631f3c0e6ceb74a5f98afc617b54785d28

Now I feel this extra check "if 'unpack' not in provider.unpack:"
before doing the automatic version
suffix striping "i = i.split('?')[0]" doesn't make sense as we'll
likely never meet a file
with a '?' in his name in a innoextract archive.

http://anonscm.debian.org/cgit/pkg-games/game-data-packager.git/tree/game_data_packager/build.py#n1123

Post by Simon McVittie
* a bug in the g-d-p code: after that has happened, the g-d-p code does
not recover gracefully, but instead tries to extract the same files
from the same archive in order to provide some other file that might
not actually be found. We should extract each archive at most once.

I've seen some useless looping before, but it only show up if there's
some other bug.
Downloading the same file in loop from GOG is not a nice behaviour,
so there's the 'lgogdownloaded' set to guard against that;

something similar is needed in fill_gap().

Alexandre

[vv221]

No more multiple calls to innoextract since commit
d6d0d7451020be2fb632fc86ea82b3b207f8b9b6 (quake3: fix
Help/DedicatedServer.html alternative)
The build now goes fine from beginning to end with the GOG installer.

Well done Alexandre ;)

[Alexandre Detiste]

Post by vv221
No more multiple calls to innoextract since commit
d6d0d7451020be2fb632fc86ea82b3b207f8b9b6 (quake3: fix
Help/DedicatedServer.html alternative)
The build now goes fine from beginning to end with the GOG installer.

Thanks for testing.

Even the HTML pages are mostly ok.

$ apt-get install linkchecker
$ linkchecker /usr/share/doc/quake3-data/Help/Index.html
564 links in 195 URLs checked. 0 warnings found. 4 errors found
/usr/share/doc/quake3-data/Help/Support/CustomerSupport/Customer_Support.htm
/usr/share/doc/quake3-data/Help/Support/Information/Minimum_System_Requirements.htm
/usr/share/doc/quake3-data/Help/Support/Information/Improving_Performance.htm
/usr/share/doc/quake3-data/Help/Licenses.txt

CustomerSupport.htm links to ~100 tiny files with mail adress of mostly defunct companies,
Minimum_System_Requirements.htm is lower than current Debian's minimum requirement.

The 4 files were all direclty linked from Index.html; maybe writing an alternate one
that links to more current information would be the way to go ;-)


---

Innoextract support should be "good enough" for normal users, now I feel it could still
be further optimized & not unpack archive we later decide we don't need...
(but in real life, users just wouldn't have downloaded all those dubs in the first time)

Maybe an extra 'UNPACKING_NEEDED' status and do the unpacking
just-in-time in the second step like the downloads are handled.

Also making the "%s should have provided %s but did not" error
more generic and not limited to .zip archive and tarballs.



if should_provide:
for missing in sorted(should_provide):
logger.error('%s should have provided %s but did not',
name, missing)

class FillResult(Enum):
UNDETERMINED = 0
IMPOSSIBLE = 1
DOWNLOAD_NEEDED = 2
COMPLETE = 3
UPGRADE_NEEDED = 4

./run simon1 -d /tmp --no-compress --no-search ../../gdp/simon
INFO:game-data-packager.build:identifying ../../gdp/simon/setup_simon_the_sorcerer_french_2.1.0.21.exe
INFO:game-data-packager.build:identifying ../../gdp/simon/setup_simon_the_sorcerer_german_2.0.0.18.exe
INFO:game-data-packager.build:identifying ../../gdp/simon/setup_simon_the_sorcerer_spanish_2.0.0.18.exe
INFO:game-data-packager.build:identifying ../../gdp/simon/setup_simon_the_sorcerer_italian_2.0.0.18.exe
INFO:game-data-packager.build:identifying ../../gdp/simon/setup_simon_the_sorcerer_2.0.0.18.exe
INFO:game-data-packager.build:identifying /tmp/gdptmp.cuyio0dr/tmp/setup_simon_the_sorcerer_2.0.0.18.exe.d/app/SIMON.WAV
INFO:game-data-packager.build:identifying /tmp/gdptmp.cuyio0dr/tmp/setup_simon_the_sorcerer_italian_2.0.0.18.exe.d/app/SIMON.VOC
INFO:game-data-packager.build:identifying /tmp/gdptmp.cuyio0dr/tmp/setup_simon_the_sorcerer_spanish_2.0.0.18.exe.d/app/SIMON.VOC
INFO:game-data-packager.build:simon-the-sorcerer1-dos-fr-data can be downloaded with lgogdownloader
INFO:game-data-packager.build:identifying /tmp/gdptmp.cuyio0dr/tmp/setup_simon_the_sorcerer_german_2.0.0.18.exe.d/app/SIMON.mp3
INFO:game-data-packager.build:identifying /tmp/gdptmp.cuyio0dr/tmp/setup_simon_the_sorcerer_french_2.1.0.21.exe.d/app/SIMON.VOC
INFO:game-data-packager.build:will not produce "simon-the-sorcerer1-dos-en-data" because better version "simon-the-sorcerer1-en-data" is also available
INFO:game-data-packager.build:will not produce "simon-the-sorcerer1-dos-fr-data" because better version "simon-the-sorcerer1-fr-data" is also available
INFO:game-data-packager.build:will not produce "simon-the-sorcerer1-dos-de-data" because better version "simon-the-sorcerer1-de-data" is also available
INFO:game-data-packager.build:will not produce "simon-the-sorcerer1-de-data" because "de" is not in LANGUAGE selection
INFO:game-data-packager.build:will not produce "simon-the-sorcerer1-it-data" because "it" is not in LANGUAGE selection
INFO:game-data-packager.build:will not produce "simon-the-sorcerer1-es-data" because "es" is not in LANGUAGE selection
INFO:game-data-packager.build:will not produce "simon-the-sorcerer1-en-data" because "fr" is prefered language
INFO:game-data-packager.build:generating package simon-the-sorcerer1-fr-data
generated "/tmp/simon-the-sorcerer1-fr-data_43_all.deb"