Gaming databases and ST Metadata

All about ST/STE games

Moderators: simonsunnyboy, Mug UK, Doctor Bob Gordon, ICS, Moderator Team

Zenichiro
Atari User
Atari User
Posts: 38
Joined: Wed Jan 04, 2006 4:20 pm
Location: Liverpool

Gaming databases and ST Metadata

Postby Zenichiro » Wed Mar 22, 2017 10:09 pm

I've recently had a bit more free time and I've been getting back into emulation. Having setup the Launchbox front-end one thing I noticed is that ST metadata is sorely lacking in both Launchbox's own and the mainstream game databases.

For example, thegamesdb.net and launchbox's database each list less than 150 ST games and rarely have more than the title and a box scan. IGDB has about 700 with similar kinds of info, although I'm a bit dubious of the accuracy - apparently Dungeon Master came out on the ST two years after the Amiga!

Mobygames does list 2376 games and has some decent information, but apparently isn't very scraper friendly and their fledgling API access seems to be by request.

Gamesdatabase.org lists 1407, has very basic information, doesn't have an API and isn't scraper friendly.

EmuMovies has zero videos of the ST apart from an example file.

Atari Mania of course lists 5617 items, including software and is fantastic, it has lots of great information, but I don't believe it has an API for retrieving metadata? Nor do I think it is suitable for (or wants) scraping.

We do have GameBase ST, which is great in what it is, a combined front-end and database. It also has a good level of completeness in what it is, but that hasn't been maintained for a long while.

Guardians of the PaST was amazing but sadly died before it got a chance to get off the ground. Atari Legend seems to be down now too? I thought I saw that up fairly recently.

So my thoughts are to pick a public database, probably thegamesdb.net as it seems to be the most 'open', and try to populate it with information about ST games, box scans and screenshots. My hope is that this is useful for those with an interest in using front ends and general ST emulation, but also to save this stuff for posterity.

I can fire up an emulator and produce the required screenshots and perhaps some videos too but I don't have a massive collection of boxed games so I'm hoping that some people who have produced scans for previous projects would be able to provide these, either to me or join me in populating one of these databases.

Any thoughts?

User avatar
Marakatti
Atari God
Atari God
Posts: 1309
Joined: Sat Jun 18, 2005 9:58 am
Location: Finland
Contact:

Re: Gaming databases and ST Metadata

Postby Marakatti » Thu Mar 23, 2017 2:55 pm

Hi, you are free to use Atarimania ST scans if you like :)
-------------< Member of Atarimania >-----------
-< ST / STe / Falcon030 / TT030 archiver >-
-------------> www.atarimania.com <-------------

User avatar
ST Graveyard
Obsessed with homicide
Obsessed with homicide
Posts: 2022
Joined: Fri Apr 26, 2002 11:53 pm
Location: Belgium
Contact:

Re: Gaming databases and ST Metadata

Postby ST Graveyard » Thu Mar 23, 2017 11:43 pm

Hello,

I've been hard at work on AtariLegend 2.0 for the past 2 years. I agree with you that an open source DB is sorely missing in the community and that is part of what will happen wit the new Atarilegend. I have still a lot of work to do, but I'm planning on building several API's for all content available on the site. Currently, even the old site is down, but we have over 4000 titles in our DB.

This all being said ... An API for the Atarimania site would be a dream ... but I've said this enough in the past so I will shut up ;-)

Cheers,
Maarten
Atari Legend IS back!

User avatar
Marakatti
Atari God
Atari God
Posts: 1309
Joined: Sat Jun 18, 2005 9:58 am
Location: Finland
Contact:

Re: Gaming databases and ST Metadata

Postby Marakatti » Fri Mar 24, 2017 8:19 am

ST Graveyard wrote:This all being said ... An API for the Atarimania site would be a dream ... but I've said this enough in the past so I will shut up ;-)


From my behalf you are free to use anything from Atarimania ST. It's just that i don't have an access to any of the code or knowledge how those things even work. Only our site owner has. I just fill the database with our BackOffice application and try to do the research that facts are correct as fast as i can :)

13.4. is my ninth year in a row i've worked with the database every week and almost daily in fact.

I still think that an universal ST/Falcon database would be ace thing! :cheers:
-------------< Member of Atarimania >-----------
-< ST / STe / Falcon030 / TT030 archiver >-
-------------> www.atarimania.com <-------------

AtariZoll
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 2795
Joined: Mon Feb 20, 2012 4:42 pm
Contact:

Re: Gaming databases and ST Metadata

Postby AtariZoll » Fri Mar 24, 2017 12:29 pm

Most of databases around, online using MySQL . So, we actually have it somehow universal already. When have standard database, you can do whatever SW for browsing, updating it - for online, or for local usage. So, I think that not the API is what is missing here - someone experienced can do it relative fast, but database self is what should be focus - doing something really rich, accurate is what one person can not do in some reasonable time. Some coordinated work only.
Would be good to have some master database for Atari ST-Falcon serial, online - with own browser, but also make database available for DL, so can be used with emulators for instance.
Negative feedback has usually positive effect.

Zenichiro
Atari User
Atari User
Posts: 38
Joined: Wed Jan 04, 2006 4:20 pm
Location: Liverpool

Re: Gaming databases and ST Metadata

Postby Zenichiro » Fri Mar 24, 2017 3:20 pm

Marakatti wrote:Hi, you are free to use Atarimania ST scans if you like :)


Hi Marakatti, thanks, that will certainly be a great help, although is there any way for me to get an archive of all the box scans? It'd be a lot quicker than downloading one by one.

I'm holding off a little on starting to add things to thegamesdb.net as there is a bit of a question mark in how well it is being maintained, I'm going to look into this a bit further. I'd hate to do a bunch of working adding games and uploading stuff only for the site to not be as useful as I thought.

For anyone familiar with Kodi, I was hoping this site could be a useful resource for emulation in the way that thetvdb.com and themoviedb.org are for media viewing.

Zenichiro
Atari User
Atari User
Posts: 38
Joined: Wed Jan 04, 2006 4:20 pm
Location: Liverpool

Re: Gaming databases and ST Metadata

Postby Zenichiro » Fri Mar 24, 2017 3:24 pm

ST Graveyard wrote:Hello,

I've been hard at work on AtariLegend 2.0 for the past 2 years. I agree with you that an open source DB is sorely missing in the community and that is part of what will happen wit the new Atarilegend. I have still a lot of work to do, but I'm planning on building several API's for all content available on the site. Currently, even the old site is down, but we have over 4000 titles in our DB.

This all being said ... An API for the Atarimania site would be a dream ... but I've said this enough in the past so I will shut up ;-)

Cheers,
Maarten


Hi Maarten, that all sounds great and a reborn Atari Legend with easy API access would be amazing, especially as it gives a lot more flexibility in platform specific information, but I do think there is also a role to play for a multi-platform resource, which may have the benefit of having a bit more stability and hopefully by its nature be less dependent on fewer key holders.

Zenichiro
Atari User
Atari User
Posts: 38
Joined: Wed Jan 04, 2006 4:20 pm
Location: Liverpool

Re: Gaming databases and ST Metadata

Postby Zenichiro » Fri Mar 24, 2017 3:52 pm

AtariZoll wrote:Most of databases around, online using MySQL . So, we actually have it somehow universal already. When have standard database, you can do whatever SW for browsing, updating it - for online, or for local usage. So, I think that not the API is what is missing here - someone experienced can do it relative fast, but database self is what should be focus - doing something really rich, accurate is what one person can not do in some reasonable time. Some coordinated work only.
Would be good to have some master database for Atari ST-Falcon serial, online - with own browser, but also make database available for DL, so can be used with emulators for instance.


That is the real kicker though, the information is all there in various databases and archives, but it needs to be accessible - none of the current sites - which is really down to just Atari Mania right now, can offer that open, automated accessibility at the moment.

When writing my original post I was trying to be sure that I didn't in any way sound critical or ungrateful for all the work and the sites that have been produced over the years by those who have put so much into the community, I also didn't want to be in the position of volunteering other peoples time and hard work.

Sadly I don't have any coding skills, nor graphical skills, so my small contribution will have to be limited to some basic grunt work.

As you've stated, the real hard work comes in adding really rich information, such as some genuinely unique and interesting prose to each game or software title and that definitely will require a lot of team work to get done on a large scale.

User avatar
ST Graveyard
Obsessed with homicide
Obsessed with homicide
Posts: 2022
Joined: Fri Apr 26, 2002 11:53 pm
Location: Belgium
Contact:

Re: Gaming databases and ST Metadata

Postby ST Graveyard » Sat Mar 25, 2017 9:28 am

I love this thread. People, please read this, let me explain something.

Around christmas 2015, at STNICCC event, I spoke with other ST fans about what was, in our opinion, missing in the ST scene. We came to the conclusion that an open source 'mother of all' databases is something that is sorely lacking. In the past, as mentioned above, so many great projects started, but died, because people always reinvented the wheel and had to add their own database entries. In 2009, Atarilegend died because the coders had enough of it, so lots of the work went over to Atarimania as well.
So last year, together with Makaratti as well, we tried to get a project of the ground to start building this open source DB. But in the end, we just had too much work with other projects (myself with the new Atarilegend), and the project got canned.
At that point, I had a mission. I want the new Atarilegend to become the 'gamesdb.net' or the 'tvdb.net' of the atari ST scene. I've been working on this project now for over 2 years together with Silver Surfer and Brume (we all have children now and lots of work, so it takes time ;-))

I'm working on the actual control panel of the site, the DB portal if you will. It takes time, with all new technologie, but everything is responsive, meaning that you can even use this control panel on your phone (it looks a bit like an app as well). I'v been spending so much efforts in this part of the project alone, that, if it is up to me, I want everybody to be able to use it, not just a team of DB people. (I do have to discuss this with the other guys of course, but I think this is the way to go ;-) )
If it is up to me I want every single ST project linked with each other. I want all the work I do to be accessible and available to as many people as possible, in all different kinds of ways (API, mysql download, browsing ... whatever). That is the only way the data can live forever, even after a project dies.

So now I'm thinking about how to do all this. My idea was building API's (which I think would be a great start). But if more can be done, let me know. I need your ideas. Atarilegend will be back anyway, in a very big way for my part. And I want to help all other projects and my work and the work of others to be used and enjoyed as much as possible. Our DB is now 15 years old, just the thought that all this work has not gone to waste and will be available to everyone, gives me hope!

As an example, take DEMOZOO. This site uses API's as well. And I'm planning to work closely together at one point when the demo section of AL comes alive. And don't know how yet, and I don't want to do all the work Makaratti has done in the past manually again in another open source DB, but one way or the other, it will get there :-) It is a sensitive topic in some way though ... And I don't want to step on people's toes...

So there we are people, let me know what you think. As mentioned, I would love more ideas, opinions ...

Currently, I have a first release of the actual new AL planned for summer 2017. Still a lot of work ... A LOT ... and it will be a release in different phases ...

Cheers,
Maarten

PS : Zenichiro, I'm happy to share any data (screenshots, boxscans (zip files) or mysql DB data) for your project. Keep in mind the current state of the AL db regarding games is by far not as good as the one from Mania. But if you need anything, let me know.
Atari Legend IS back!

penguin
Obsessive compulsive Atari behavior
Obsessive compulsive Atari behavior
Posts: 108
Joined: Tue Dec 24, 2013 10:43 am

Re: Gaming databases and ST Metadata

Postby penguin » Sat Mar 25, 2017 2:04 pm

For what it's worth, you may use the images from AtariUpToDate as well. Not that many game screenshots compared to AM, but still some pics (3200).

My site (which has 9310 apps listed) doesn't have an API either, although it's not a big problem to create one.
AtariUpToDate - Atari ST/TT/Falcon software database and version tracker: http://www.atariuptodate.de
st-computer magazine - http://st-computer.atariuptodate.de/

User avatar
ST Graveyard
Obsessed with homicide
Obsessed with homicide
Posts: 2022
Joined: Fri Apr 26, 2002 11:53 pm
Location: Belgium
Contact:

Re: Gaming databases and ST Metadata

Postby ST Graveyard » Sat Mar 25, 2017 6:26 pm

Wow, I had never seen Atariuptodate. I would be interested in an API. We could work together at one point. Comparing the AL database with your database by use of interface. It can be a WIN - WIN ...
Those are the things I'm talking about ...
Atari Legend IS back!

User avatar
Stefan jL
Atari God
Atari God
Posts: 1234
Joined: Thu May 09, 2002 3:21 pm
Location: Sweden
Contact:

Re: Gaming databases and ST Metadata

Postby Stefan jL » Sat Apr 29, 2017 10:49 am

ST Graveyard can you post a screenshot showing how an ST game entry will look like on the new AL just to make it easier to come with opinions and ideas... i mean what info is there for each entry and such?
Image

mikro
Atari God
Atari God
Posts: 1282
Joined: Sat Sep 10, 2005 11:11 am
Location: Brisbane, Queensland, Australia
Contact:

Re: Gaming databases and ST Metadata

Postby mikro » Sun Apr 30, 2017 2:33 am

It's great to see some progress in this area. Frankly, I still don't get the need for something new (AL, AL 2.0, "super db", ...) instead of expanding the current one (Atari Mania) -- why isn't possible to ask the owner for source/db and fork from there? You can add all sorts of APIs, responsiveness etc.

User avatar
LynXX
Retro freak
Retro freak
Posts: 13
Joined: Wed Jul 19, 2017 3:15 am
Location: Seattle, WA, USA

Re: Gaming databases and ST Metadata

Postby LynXX » Mon Aug 07, 2017 1:10 am

I think we all agree that a database is needed, some kind of open-source repository of the data itself. Having websites around it is also key, but it's kind of a chicken and the egg problem: No website to enter the data = no database, and not good enough database = no incentive to make a website public / available.

So I wonder if we should focus on the data itself, while websites (the new AtariLegend or others) are built in parallel. One way to do that would be to store the data in standard text files (XML, SQL dumps, ...) into a GitHub repository for instance. Then anybody can (reasonably) easily contribute to it via pull requests with just a text editor. The starting point could be an export from what would be considered the "best" database at the moment. I guess even scripts could be written to try to merge the data from multiple sources.

This would be similar to what's done with "dat files" for MAME or other emulation platforms, or what TOSEC is doing, except the files would not only contain just a list of games but all the detailed information we want (publishers, developers, artists, if the game appear on a menu disc, magazine reviews, etc.)

I feel that would allow us to make progress on an authoritative list of games / software, while keeping it open source & available, and on top of which great websites could be built. One concern with the website-only approach is that the data is never made available, so when the website goes down everything is lost and we have to start again...

The nice property with this is that the data becomes "universal", and can be easily converted to any format like a MySQL database for website applications, or something else for desktop apps (GamebaseST) or emulator frontends.

User avatar
ST Graveyard
Obsessed with homicide
Obsessed with homicide
Posts: 2022
Joined: Fri Apr 26, 2002 11:53 pm
Location: Belgium
Contact:

Re: Gaming databases and ST Metadata

Postby ST Graveyard » Mon Aug 07, 2017 4:55 pm

Hello all,

I agree with everybody. Here is what I'm doing regarding Atari Legend 2.0 . I hope to have a first release ready by the end of september 2017. This release will have :
- a game section with already possibility to download parts of the database in excel, CSV or JSON
- reviews, interviews and all content of the old site
- completely open control panel

So regarding this last point. I have been working on an admin section for the past 2 years. This is the part of the project where we enter data in the db. I'm am planning to open this one for the public. If you create an account, you can see the complete control panel in view mode. At control panel level (and DB structure level), the menu section, download section, demo section are already finished. The database structure is completely normalised and modular. The download section and menu sections are massive! But everything can be expanded.
I'm also releasing the complete technical documentation with all db structure, so you can see how everything is linked together.

At a later release (perhaps the second release by the end of this year) I'm planning an API (so I'm definitely looking for people with other projects to test this stuff) and download functions to fully download the database and website content to make it all open to the public.
I also like to open up the GITHUB at one point in time, so even all the php code becomes public, a bit like the DEMOZOO project (if people are interested to help out with the code, let me know).

We are also planning a complete merge with STONISH.NET to link all games with menu's, hackers, crackers ... and all that stuff. I also would love to play with the DEMOZOO API and with all the FUJIOLOGY content to create the ultimate opensource DB.

Lot's of ambition. But first I will focus and the september release as I am becoming a bit nervous, still lots of work to do.

DB content wise, it is obvious that Atari Mania is the best ST database on the planet, but I have no control over this and I am also not planning to do all the work that has been done at AM over again at AL. I will update the database by use of other sources (eg. the new MOBI GAMES API), but I personally will not do these kinds of things manually anymore as this is not my ambition and I am partly creating this project to simply avoid that kind of work in the first place). The database will be updated, but I keep my fingers crossed for maybe, hopefully, some time, who knows ... An AM API! ;-)

So yes, I do think the whole excel base mentioned above is a cool idea and I would even donate the DB content we have now BUT if you give me 1.5 months you can download it from the site!

Conclusion, I do think we are all on the same page here, I am planning to create a fully open source (as open as possible) project as I think this is sorely missing in the scene. But for september, we will have to do with the AL games DB (which is still pretty big (4500+ games, 20 000 screenshots, a massive amount of publishers, developpers, individuels, reviews, interviews ... ). I am open to all suggestions to make to project bigger, better and more fun when it is released.

Stefan LJ, I am sorry I did not see your reply. PM what you want to know, which structures you like to see, and I will sent it to you. But like I said, it will be available soon...

For now, we have this : www.atarilegend.com ;-)

Cheers,
Maarten
Atari Legend IS back!

User avatar
calimero
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 2046
Joined: Thu Sep 15, 2005 10:01 am
Location: STara Pazova, Serbia
Contact:

Re: Gaming databases and ST Metadata

Postby calimero » Tue Aug 08, 2017 3:10 am

AtariZoll wrote:Would be good to have some master database for Atari ST-Falcon serial, online - with own browser, but also make database available for DL, so can be used with emulators for instance.

How jookie Comsosex option of "Downloading of floppy images using simple tool from internet directly to the device (something similar to some app store)" works?

From what URLs are MSA files downloaded?
It would be cool if there were some more information about every game before download...
using Atari since 1986.http://wet.atari.orghttp://milan.kovac.cc/atari/software/ ・ Atari Falcon030/CT63/SV ・ Atari STe ・ Atari Mega4/MegaFile30/SM124 ・ Amiga 1200/PPC ・ Amiga 500 ・ C64 ・ ZX Spectrum ・ RPi ・ MagiC! ・ MiNT 1.18 ・ OS X

User avatar
ST Graveyard
Obsessed with homicide
Obsessed with homicide
Posts: 2022
Joined: Fri Apr 26, 2002 11:53 pm
Location: Belgium
Contact:

Re: Gaming databases and ST Metadata

Postby ST Graveyard » Tue Aug 08, 2017 7:50 am

calimero wrote:
AtariZoll wrote:Would be good to have some master database for Atari ST-Falcon serial, online - with own browser, but also make database available for DL, so can be used with emulators for instance.

How jookie Comsosex option of "Downloading of floppy images using simple tool from internet directly to the device (something similar to some app store)" works?

From what URLs are MSA files downloaded?
It would be cool if there were some more information about every game before download...


Calimero, Brume is working closely together with jookie to completely link Stonish.net with the Cosmos tool. Brume is part of AL. Stonish db will be merged with AL. Hence Cosmos will be linked to complete open AL db.

The sky is the limit, people! I just need some more time to finish the first release of the project.

Brume, correct me if this info is incorrect.

Cheers...
Atari Legend IS back!

User avatar
calimero
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 2046
Joined: Thu Sep 15, 2005 10:01 am
Location: STara Pazova, Serbia
Contact:

Re: Gaming databases and ST Metadata

Postby calimero » Tue Aug 08, 2017 3:19 pm

ST Graveyard wrote:Calimero, Brume is working closely together with jookie to completely link Stonish.net with the Cosmos tool. Brume is part of AL.

First we need "people" (aka Atari Sceners) database to know who is who :D

---
My two cents regarding "databases":

we have "data" (in our case TOS executable, games or apps) and we have "metadata" (informations about it).

I choose approach explained here: http://milan.kovac.cc/atari/software/about/about.php

I use simple .txt file for holding metadata for each game/app.
Metadata are in .txt file contain key:value pairs so you can simple add whatever you need for specific game/app.
Screenshots/videos are stored in same directory as game/app.
For editing and manipulating you can use simple text editor (or any tool/framework/library for manipulating strings) and any file manager.

You can also use "website" that I create for browsing, manipulating and searching data and metadata (which also provide version control): http://milan.kovac.cc/atari/software/

I prefer this approach since every computer (and every user) have ability to manipulate with strings and files.
Give RAW data and let people use it as they please.
Yes, I know: you do not have "built-in" relations like in real database but you can build these relations from information inside .txt files.

It is philosophical and practical matter in same time (recommendation: https://www.youtube.com/watch?v=zumdnI4EG14)


In lack of good universal file manger today (Windows Explorer, Finder, Nautilus, WindowsCommander...) people start to build special "browsers" (applications) for certain kind of files (on Mac: iTunes, Photos...) to be able to manipulate, sorting, searching, browsing... this certain kind of files with "style" :D
In past decades we allow terrible, terrible situation of manipulating, combining, comparing, searching, managing... of ANY data on computers. (Please do watch youtube that I post above)
using Atari since 1986.http://wet.atari.orghttp://milan.kovac.cc/atari/software/ ・ Atari Falcon030/CT63/SV ・ Atari STe ・ Atari Mega4/MegaFile30/SM124 ・ Amiga 1200/PPC ・ Amiga 500 ・ C64 ・ ZX Spectrum ・ RPi ・ MagiC! ・ MiNT 1.18 ・ OS X

User avatar
ST Graveyard
Obsessed with homicide
Obsessed with homicide
Posts: 2022
Joined: Fri Apr 26, 2002 11:53 pm
Location: Belgium
Contact:

Re: Gaming databases and ST Metadata

Postby ST Graveyard » Wed Aug 09, 2017 9:19 pm

Calimero,

But why would you do all this into txt files when you can have a fully normalized and modular table structure in SQL, which you can easily export and manipulate in any way and into any form you wish? Even pure ascii txt files ...

Maarten
Atari Legend IS back!

User avatar
calimero
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 2046
Joined: Thu Sep 15, 2005 10:01 am
Location: STara Pazova, Serbia
Contact:

Re: Gaming databases and ST Metadata

Postby calimero » Thu Aug 10, 2017 10:52 am

I prefer using RAW data (bytes in files) then "hidden" data in SQL base (and therefor, to use SQL tools and rules).

SQL give benefactions but also add complexity. I do not see need to use SQL in this particular case. Data on my site can be manipulate over web interface (website) that I build but they can be also manipulated on any computer, including ST with existing tools (text editor and file manager). Also others can use data without any conversion - all data are in "RAW" format.

http://milan.kovac.cc/atari/software/rsync/ - you can browse data in any browser without any HTML/CSS/JS decoration ;)



btw
this talk remind me on Microsoft WinFS - Microsoft idea to use SQL-like database as filesystem in OS. Good idea but they never implement it. (and if you look what mess they made with Registry instead of .conf files... maybe it is better that WinFS was never implemented :D)

I would prefer something better than SQL, maybe these relatively new noSQL database, but with nice GUI (I also keep all data in key:value pairs as I believe that it is most appropriate form of storing data).

We still did not tackle problem of using computer for data manipulation in appropriate way.
Ted Nelson present ZigZag structure: https://www.youtube.com/watch?v=WEj9vqVvHPc
I would like to see at least meaning of links:

Image

From this image you see what John Methieson influenced on. You see where he meet Martin Brennan. When, what and how they do... (idea is to be able to zoom in and out to see more details if you like).

Sorry for stretching topic but I am really interested in data storing and retrieving :)
using Atari since 1986.http://wet.atari.orghttp://milan.kovac.cc/atari/software/ ・ Atari Falcon030/CT63/SV ・ Atari STe ・ Atari Mega4/MegaFile30/SM124 ・ Amiga 1200/PPC ・ Amiga 500 ・ C64 ・ ZX Spectrum ・ RPi ・ MagiC! ・ MiNT 1.18 ・ OS X


Social Media

     

Return to “Games - General”

Who is online

Users browsing this forum: No registered users and 1 guest