MiSTer updater script

https://github.com/MiSTer-devel/Main_MiSTer/wiki

Moderators: Mug UK, Zorro 2, Greenious, spiny, Sorgelig, Moderator Team

Locutus73
Atari Super Hero
Atari Super Hero
Posts: 537
Joined: Wed Feb 07, 2018 6:13 pm

Re: MiSTer updater script

Postby Locutus73 » Sun Dec 16, 2018 10:33 pm

Version 1.3.3 released

Version 1.3.3 - 2018.12.16 - Updating the bootloader before deleting linux.img, moved the Linux system update at the end of the script with an "atomic" approach (first extracting in a linux.update directory and then moving files).
Version 1.3.2 - 2018.12.16 - Deleting linux.img before updating the linux directory so that the extracted new file won't be overwritten.
Version 1.3.1 - 2018.12.16 - Disabled Linux updating as default behaviour.
Version 1.3 - 2018.12.16 - Added Kernel, Linux filesystem and bootloader updating functionality; added autoreboot option.
Version 1.2 - 2018.12.14 - Added support for distinct directories for computer cores, console cores, arcade cores and service cores; added an option for removing "Arcade-" prefix from arcade core names
Version 1.1 - 2018.12.11 - Added support for additional repositories (i.e. Scaler filters and Game Boy palettes), renamed some variables
Version 1.0 - 2018.12.11 - First commit

https://github.com/MiSTer-devel/Updater_script_MiSTer

Can some brave volunteer test the whole Linux update uncommenting
SD_INSTALLER_URL="https://github.com/MiSTer-devel/SD-Installer-Win64_MiSTer"
please?
It's experimental and can lead to a non booting SD.
If MiSTer doesn't boot, try to eject the SD card and, from a PC, open open release_20181109.rar and extract just the content of files/linux/ into linux dir on SD; then reflash the bootloader; it's not sure 100% it will boot, but it should...
...again: it's experimental, don't blame me if your DE10-Nano catches fire.

Please report back both if it works or if you encounter problems.

Thank you in advance.
Best regards.

Locutus73

SaschaFFM
Atari User
Atari User
Posts: 42
Joined: Mon Feb 05, 2018 8:24 am

Re: MiSTer updater script

Postby SaschaFFM » Mon Dec 17, 2018 9:25 am

I have just updated to the latest version 1.3.3 and uncommented the line.

Code: Select all

# ./mister_update.sh
Checking https://github.com/MiSTer-devel/SD-Installer-Win64_MiSTer
Downloading https://github.com/MiSTer-devel/SD-Installer-Win64_MiSTer/blob/master/release_20181109.rar?raw=true
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   155    0   155    0     0    284      0 --:--:-- --:--:-- --:--:--   286
100   166  100   166    0     0    188      0 --:--:-- --:--:-- --:--:-- 41500
100 60.6M  100 60.6M    0     0  1615k      0  0:00:38  0:00:38 --:--:-- 2861k

Checking https://github.com/MiSTer-devel/Main_MiSTer
Nothing to update
(...)


All went through successfully. Restart of the MiSTer went well. All is working just fine for me!

Love your script. It is making usage of the MiSTer so much easier.

esmith13
Atari User
Atari User
Posts: 32
Joined: Tue Oct 30, 2018 8:06 pm
Contact:

Re: MiSTer updater script

Postby esmith13 » Mon Dec 17, 2018 11:29 am

I used v1.3.3 this morning via PuTTY using your command to pull and run the latest updater script.

(Keep in mind my SD card has been built based on the SmokeMonster MiSTer build script)
Here are a few items of note from my results:
    * New Arcade Cores Bubbles, Splat and StarGate were not added but Robotron and Joust were.
    * Atari 800 core was placed in both _Consoles and _Computers. (I know it's related to how the github is setup but you could resolve the issue in your script since it is a known constant)
    * Any core that is updated will have all previous versions of that same core deleted if deletions are on. Even custom builds of the core with different names. I understand why this is but it doesn't account for those of us that still have nearest neighbor versions of cores (_NN at end of name)

All in all I love this script and thank you very much for it!! The list above shouldn't be taken as "complaints" just pointing out a few things you may not have thought of or planned for. I have workarounds I can do on my end to prevent issues and still benefit from your script. Today was also a great day to try it out as Sorgelig has been an absolute MACHINE with releases this weekend!!

Thank you again for this wonderful script!
And thank you Sorgelig for being an inhuman BEAST of a machine with updates and new releases!!!

Locutus73
Atari Super Hero
Atari Super Hero
Posts: 537
Joined: Wed Feb 07, 2018 6:13 pm

Re: MiSTer updater script

Postby Locutus73 » Mon Dec 17, 2018 5:41 pm

SaschaFFM wrote:I have just updated to the latest version 1.3.3 and uncommented the line.
[...]
All went through successfully. Restart of the MiSTer went well. All is working just fine for me!

Love your script. It is making usage of the MiSTer so much easier.

Good, good, thank you for your feedback.

esmith13 wrote:[New Arcade Cores Bubbles, Splat and StarGate were not added but Robotron and Joust were.

Bubbles core name is Arcade-Bubbles.rbf and it lacks any timestamp like other cores and I cannot parse it.
Splat and StarGate are in a "Release" directory, not "releases", I will easily fix this.

esmith13 wrote:Atari 800 core was placed in both _Consoles and _Computers. (I know it's related to how the github is setup but you could resolve the issue in your script since it is a known constant)

Well, https://github.com/MiSTer-devel/Atari800_MiSTer is linked both by Atari 800XL computer core and Atari 5200 console core in the Wiki, so it ends both in _Computer and _Console directories; it seems to me a fair result.

esmith13 wrote:Any core that is updated will have all previous versions of that same core deleted if deletions are on. Even custom builds of the core with different names. I understand why this is but it doesn't account for those of us that still have nearest neighbor versions of cores (_NN at end of name)[/list]

Ok, I think I can easily solve this issue.

esmith13 wrote:All in all I love this script and thank you very much for it!! The list above shouldn't be taken as "complaints" just pointing out a few things you may not have thought of or planned for. I have workarounds I can do on my end to prevent issues and still benefit from your script. Today was also a great day to try it out as Sorgelig has been an absolute MACHINE with releases this weekend!!

Don't get me wrong, I must thank you for the feedback and the bug report, this isn't a "complaints" list, on the contrary, it's a precious help.

esmith13 wrote:Thank you again for this wonderful script!
And thank you Sorgelig for being an inhuman BEAST of a machine with updates and new releases!!!

Thank you and thanks to Sorgelig.

Regards.
Locutus73

Sorgelig
Ultimate Atarian
Ultimate Atarian
Posts: 5826
Joined: Mon Dec 14, 2015 10:51 am
Location: Russia/Taiwan

Re: MiSTer updater script

Postby Sorgelig » Mon Dec 17, 2018 6:01 pm

Locutus73 wrote:Bubbles core name is Arcade-Bubbles.rbf and it lacks any timestamp like other cores and I cannot parse it.
Splat and StarGate are in a "Release" directory, not "releases", I will easily fix this.

I will fix the paths.

esmith13
Atari User
Atari User
Posts: 32
Joined: Tue Oct 30, 2018 8:06 pm
Contact:

Re: MiSTer updater script

Postby esmith13 » Mon Dec 17, 2018 6:03 pm

Sorgelig wrote:
Locutus73 wrote:Bubbles core name is Arcade-Bubbles.rbf and it lacks any timestamp like other cores and I cannot parse it.
Splat and StarGate are in a "Release" directory, not "releases", I will easily fix this.

I will fix the paths.


Thank you!!!!

For this and EVERYTHING!!!! :D

Locutus73
Atari Super Hero
Atari Super Hero
Posts: 537
Joined: Wed Feb 07, 2018 6:13 pm

Re: MiSTer updater script

Postby Locutus73 » Tue Dec 18, 2018 6:10 pm

Hi @Sorgelig.
I just made a pull request for Main_MiSTer/menu.cpp.
I implemented a Scripts menu in the main System Settings page. The menu scans for all *.sh files in the SD and, if one is selected, it executes it showing the standard output in the user interface; a Cancel option allows to interrupt the script and becomes OK once the script has finished.
If you like the code, users will be able to use the updater script or any other script directly from the user interface. Obviously I will have to shorten the updater output in order to make it firendly for the user interface.

Thank you in advance.
Best regards.

Locutus73

uigiflip
Atari freak
Atari freak
Posts: 61
Joined: Sun May 13, 2018 6:41 pm

Re: MiSTer updater script

Postby uigiflip » Tue Dec 18, 2018 9:21 pm

cool so will this be a menu option in next menu update?
Replay 2, Mister, FPGA Replay + 68060 Daughterboard

SaschaFFM
Atari User
Atari User
Posts: 42
Joined: Mon Feb 05, 2018 8:24 am

Re: MiSTer updater script

Postby SaschaFFM » Tue Dec 18, 2018 9:54 pm

That would be a great addition!

BBond007
Captain Atari
Captain Atari
Posts: 414
Joined: Wed Feb 28, 2018 3:23 am

Re: MiSTer updater script

Postby BBond007 » Wed Dec 19, 2018 3:49 am

Locutus73 wrote:I just made a pull request for Main_MiSTer/menu.cpp.


I pulled those changes into my fork and ran the tested the updater from the menu.

Worked great with the exception of downgrading my Main_MiSTer which is the expected behavior in this case.

Certainly this script runner has other applications as well.

For example like a client and server scripts that connect two MiSTers together over serial for network game play. I'm thinking Lotus Turbo Challenge.

Thanks!

Sorgelig
Ultimate Atarian
Ultimate Atarian
Posts: 5826
Joined: Mon Dec 14, 2015 10:51 am
Location: Russia/Taiwan

Re: MiSTer updater script

Postby Sorgelig » Wed Dec 19, 2018 7:29 pm

Locutus73 wrote:I just made a pull request for Main_MiSTer/menu.cpp.

Good addition!

I didn't try your script yet as i use other folder structure, but will use it later.
I think you need to compare downloaded MiSTer binary and currently used one and skip the update if they are the same.
linux/kernel/uboot updater is batter to place into separate script - these parts rarely updated. Linux.img can be customized, so it's better not to include these parts into main updater script.

Locutus73
Atari Super Hero
Atari Super Hero
Posts: 537
Joined: Wed Feb 07, 2018 6:13 pm

Re: MiSTer updater script

Postby Locutus73 » Wed Dec 19, 2018 7:59 pm

Sorgelig wrote:
Locutus73 wrote:I just made a pull request for Main_MiSTer/menu.cpp.

Good addition!

Thank you

Sorgelig wrote:I didn't try your script yet as i use other folder structure, but will use it later.

The folder structure is configurable through an array
declare -A CORE_CATEGORY_PATHS=(
["cores"]="$BASE_PATH/_Computer"
["console-cores"]="$BASE_PATH/_Console"
["arcade-cores"]="$BASE_PATH/_Arcade"
["service-cores"]="$BASE_PATH/_Utility"
)

Sorgelig wrote:I think you need to compare downloaded MiSTer binary and currently used one and skip the update if they are the same.

I do this for all cores. In the special case of main MiSTer executable and menu.rbf I keep on the SD both a copy of the file with the timestamp and the one without the timestamp (i.e. both MiSTer_20181213 and MiSTer) so I use the timestamped one as a reference for determining if I have to update it or not. There is an added benefit: when you choose a core you can choose i.e. menu_20180902.rbf so you can return to the main menu from a core.

Sorgelig wrote:linux/kernel/uboot updater is batter to place into separate script - these parts rarely updated. Linux.img can be customized, so it's better not to include these parts into main updater script.

In this moment the linux/kernel/uboot updater is in the script, but it's disabled by default through a variable which is commented
#SD_INSTALLER_URL="https://github.com/MiSTer-devel/SD-Installer-Win64_MiSTer"
you have to uncomment it to actually enable the linux/kernel/uboot updater.

Regards.
Locutus73

Locutus73
Atari Super Hero
Atari Super Hero
Posts: 537
Joined: Wed Feb 07, 2018 6:13 pm

Re: MiSTer updater script

Postby Locutus73 » Wed Dec 19, 2018 8:00 pm

DOUBLE POST
DELETED
Last edited by Locutus73 on Wed Dec 19, 2018 8:59 pm, edited 1 time in total.

Sorgelig
Ultimate Atarian
Ultimate Atarian
Posts: 5826
Joined: Mon Dec 14, 2015 10:51 am
Location: Russia/Taiwan

Re: MiSTer updater script

Postby Sorgelig » Wed Dec 19, 2018 8:08 pm

Locutus73 wrote: In the special case of main MiSTer executable and menu.rbf

unlike MiSTer executable, menu.rbf is not executable, so the file is not used once it's loaded. So it's free to replace like any other core at runtime. If menu.rbf will be corrupted then anyway you won't be able to load other version as you won't have the video.

Locutus73 wrote:In this moment the linux/kernel/uboot updater is in the script, but it's disabled by default through a variable which is commented
#SD_INSTALLER_URL="https://github.com/MiSTer-devel/SD-Installer-Win64_MiSTer"
you have to uncomment it to actually enable the linux/kernel/uboot updater.

as we talking about running the script from UI, you can't modify the script. So, from practical point of view it's better to have 2 scripts. As an additional benefit, core update script will be more simple without handling rar and other OS/boot update parts.

Sorgelig
Ultimate Atarian
Ultimate Atarian
Posts: 5826
Joined: Mon Dec 14, 2015 10:51 am
Location: Russia/Taiwan

Re: MiSTer updater script

Postby Sorgelig » Wed Dec 19, 2018 8:18 pm

Locutus73 wrote:The folder structure is configurable through an array
declare -A CORE_CATEGORY_PATHS=(
["cores"]="$BASE_PATH/_Computer"
["console-cores"]="$BASE_PATH/_Console"
["arcade-cores"]="$BASE_PATH/_Arcade"
["service-cores"]="$BASE_PATH/_Utility"
)


would be good if script will check the external parameters file and use settings from there if found. In this case no need to tweak every script version.

Locutus73
Atari Super Hero
Atari Super Hero
Posts: 537
Joined: Wed Feb 07, 2018 6:13 pm

Re: MiSTer updater script

Postby Locutus73 » Wed Dec 19, 2018 8:47 pm

Sorgelig wrote:
Locutus73 wrote: In the special case of main MiSTer executable and menu.rbf

unlike MiSTer executable, menu.rbf is not executable, so the file is not used once it's loaded. So it's free to replace like any other core at runtime. If menu.rbf will be corrupted then anyway you won't be able to load other version as you won't have the video.

Anyway the update seems to work flawlessly without any file corruption, even updating the MiSTer executable or menu.rbf. As we discussed some time ago, Linux (Unix in general, so OSX too) let you move/delete open files, because it mantains the handle regardless of the directory structure. It seems solid even with exFat in MiSTer. If the script updates MiSTer or menu.rbf (or linux/kernel/uboot) it can optionally (it actually does it by default) reboot the system at the end of the update.

Sorgelig wrote:
Locutus73 wrote:In this moment the linux/kernel/uboot updater is in the script, but it's disabled by default through a variable which is commented
#SD_INSTALLER_URL="https://github.com/MiSTer-devel/SD-Installer-Win64_MiSTer"
you have to uncomment it to actually enable the linux/kernel/uboot updater.

as we talking about running the script from UI, you can't modify the script. So, from practical point of view it's better to have 2 scripts. As an additional benefit, core update script will be more simple without handling rar and other OS/boot update parts.

It's a possibility, why not, maybe we'll have a script repository in the future.

On the other side anyone could make two copies of the script with different options or even better use this trick I personally use: instead of copying the updater script on the SD and launcing it, I use an update.sh with this simple code:

Code: Select all

#!/bin/bash

curl -ksLf https://github.com/MiSTer-devel/Updater_script_MiSTer/blob/master/mister_updater.sh?raw=true | bash -
as a nice side effects this update.sh always uses the last version of the mister_updater.sh script.

Users can personalize options (and eventually make different scripts) using sed, i.e.

Code: Select all

#!/bin/bash

curl -ksLf https://github.com/MiSTer-devel/Updater_script_MiSTer/blob/master/mister_updater.sh?raw=true | sed 's/BASE_PATH\/_Arcade/BASE_PATH\/_Different_Arcade_Dir/g' | bash -
changes the _Arcade destination directory to _Different_Arcade_Dir

or

Code: Select all

#!/bin/bash

curl -ksLf https://github.com/MiSTer-devel/Updater_script_MiSTer/blob/master/mister_updater.sh?raw=true | sed 's/#SD_INSTALLER_URL/SD_INSTALLER_URL/g' | bash -
uncomments the SD_INSTALLER_URL variable actually enabling the linux/kernel/uboot update; the user could name it update_linux.sh

Or we can provide different scripts. We have limitless options and editing a script is quite simple.

P.S.: I was thinking... once the Script menu is released, in the next SD Installer you could provide a simple update.sh as described above, so the wiki Setup-Guide could be simplified.


Regards.
Locutus73
Last edited by Locutus73 on Wed Dec 19, 2018 8:55 pm, edited 3 times in total.

Locutus73
Atari Super Hero
Atari Super Hero
Posts: 537
Joined: Wed Feb 07, 2018 6:13 pm

Re: MiSTer updater script

Postby Locutus73 » Wed Dec 19, 2018 8:50 pm

Sorgelig wrote:would be good if script will check the external parameters file and use settings from there if found. In this case no need to tweak every script version.

Please read my other post, I'd use/distribute to users the simple curl|bash trick and put options there, with the benefit of automatically using always the last updater release while mantaining the customizations.

Locutus73

djsquare
Atari freak
Atari freak
Posts: 61
Joined: Thu Oct 18, 2018 2:01 am

Re: MiSTer updater script

Postby djsquare » Thu Dec 20, 2018 1:18 am

1st time posting. I just wanted to say thank you for making this script. This makes life so much easier for me. When I get home from work, I don't need to update things one by one. I really hope more people use this :cheers:

Sorgelig
Ultimate Atarian
Ultimate Atarian
Posts: 5826
Joined: Mon Dec 14, 2015 10:51 am
Location: Russia/Taiwan

Re: MiSTer updater script

Postby Sorgelig » Thu Dec 20, 2018 6:06 am

Locutus73 wrote:Please read my other post, I'd use/distribute to users the simple curl|bash trick and put options there, with the benefit of automatically using always the last updater release while mantaining the customizations.

So you ask every user to be a professional in Linux command prompt instead to manage a simple text file with options.
Probably, if MiSTer would need to use sed in order to run the cores - no one would use MiSTer.
Ok. It's up to you.

User avatar
Newsdee
Atari God
Atari God
Posts: 1539
Joined: Fri Sep 19, 2014 8:40 am

Re: MiSTer updater script

Postby Newsdee » Thu Dec 20, 2018 7:00 am

Aren't there INI parsers in shell scripts? (there are in python but that's a much heavier dependency)

Locutus73
Atari Super Hero
Atari Super Hero
Posts: 537
Joined: Wed Feb 07, 2018 6:13 pm

Re: MiSTer updater script

Postby Locutus73 » Thu Dec 20, 2018 7:36 am

Sorgelig wrote:
Locutus73 wrote:Please read my other post, I'd use/distribute to users the simple curl|bash trick and put options there, with the benefit of automatically using always the last updater release while mantaining the customizations.

So you ask every user to be a professional in Linux command prompt instead to manage a simple text file with options.
Probably, if MiSTer would need to use sed in order to run the cores - no one would use MiSTer.
Ok. It's up to you.

No, I didn’t mean users should use sed or write bash code, I’m thinking to something different, please tell me if you like it... I was thinking to make a second update.sh script with a list of simple, user friendly variables at the beginning and then short code in order to prepare and launch the curl|sed|bash command. Something like this:

Code: Select all

#/bin/bash

#User friendly variables
COMPUTER_DIR="_Computer"
CONSOLE_DIR="_Console"
ARCADE_DIR="_Arcade"
UTILITY_DIR="_Utility"
DELETE_OLD_CORES=true
UPDATE_ONLY_INSTALLED_CORES=false
UPDATE_FILTERS=true
UPDATE_GAMEBOY_PALETTES=true
UPDATE_LINUX_SYSTEM=false
[other variables if needed]

#===code user shouldn’t change===
REMOTE_UPDATE_COMMAND=""
[some code to prepare the sed commands corresponding to the user friendly variables]
REMOTE_UPDATE_COMMAND="curl -ksLf https://github.com/MiSTer-devel/Updater_script_MiSTer/blob/master/mister_updater.sh?raw=true $REMOTE_UPDATE_COMMAND | bash -"
$REMOTE_UPDATE_COMMAND
Then we should distribute to users this update.sh instead of mister_updater.sh. This script would act as the INI file you suggested and then download in RAM and execute always the last mister_updater.sh with the added benefit of being self contained instead of two separate files. Add to that the fact that users could have various different (with different parameter sets) copies of this script with different names, i.e. a regular update.sh with default settings and a update_no_filters_no_palettes.sh with UPDATE_FILTERS=false UPDATE_GAMEBOY_PALETTES=false; if we separate the script from the settings they should prepare two different settings files, two different script files and edit the script files in order to load the different settings files. This way the script and the settings are consolidated: users just edit the first user readable settings part of the script and always use the online last updater.
@Sorgelig please tell me what you think.

Thank you in advance.
Best regards.

Locutus73
Last edited by Locutus73 on Thu Dec 20, 2018 12:32 pm, edited 4 times in total.

Locutus73
Atari Super Hero
Atari Super Hero
Posts: 537
Joined: Wed Feb 07, 2018 6:13 pm

Re: MiSTer updater script

Postby Locutus73 » Thu Dec 20, 2018 7:39 am

Newsdee wrote:Aren't there INI parsers in shell scripts? (there are in python but that's a much heavier dependency)

Parsing text files is trivial using regular expressions (grep sed awk), that’s not the problem, but I proposed something slightly different in another post.

Regards.

Locutus73

Locutus73
Atari Super Hero
Atari Super Hero
Posts: 537
Joined: Wed Feb 07, 2018 6:13 pm

Re: MiSTer updater script

Postby Locutus73 » Thu Dec 20, 2018 7:57 am

djsquare wrote:1st time posting. I just wanted to say thank you for making this script. This makes life so much easier for me. When I get home from work, I don't need to update things one by one. I really hope more people use this :cheers:

Thank you, that’s was the idea: I made the script for my own personal convenience, because I am super super lazy and the idea of manually updating stuff kills me... then why not sharing?
So I wrote the script not because I’m an active guy, but because I’m lazy :lol:

Regards.
Locutus73

Doozer
Atariator
Atariator
Posts: 28
Joined: Fri Apr 22, 2016 12:26 pm

Re: MiSTer updater script

Postby Doozer » Thu Dec 20, 2018 11:33 am

Hi Locutus73,

Here is a way to handle the Linux rootfs update. I am using the following steps on Mister and never experienced an issue.

1/ copy linux.img to linux.img_new
2/ delete linux.img
3/ rename linux.img_new to linux.img
4/ reboot

With these 4 steps, the FS integrity is ensured and the OS not confused.

Cheers!

Locutus73
Atari Super Hero
Atari Super Hero
Posts: 537
Joined: Wed Feb 07, 2018 6:13 pm

Re: MiSTer updater script

Postby Locutus73 » Thu Dec 20, 2018 11:55 am

Doozer wrote:Hi Locutus73,

Here is a way to handle the Linux rootfs update. I am using the following steps on Mister and never experienced an issue.

1/ copy linux.img to linux.img_new
2/ delete linux.img
3/ rename linux.img_new to linux.img
4/ reboot

With these 4 steps, the FS integrity is ensured and the OS not confused.

Cheers!


That's basically what the 1.3.3 script does:
1) first it does some cumbersome stuff in order to download unrar-nonfree if needed, then it checks the integrity of the SD Installer RAR package and finally it extracts the linux directory in a temporary linux.update directory
2) then it moves updateboot and uboot.img to the final linux destination directory and performs a bootloader update
3) it deletes linux/linux.img (the root filesystems becomes readonly)
4) it moves the rest of the linux.update files into the final linux destination directory, it deletes the temporary linux.update directory, it syncs the filesystems and immediately reboots.

It seems to work perfectly without any file corruption, but some seconds after the deletion of the original linux.img, after the root filesystem becomes read only, if I don't immediately reboot, some commands segfaults. Now the procedure seems safe, but, in order to be supersafe, I will always keep it disabled by default, with a warning for users that want to enable it.

Regards.
Locutus73


Return to “MiSTer”

Who is online

Users browsing this forum: vanfanel and 15 guests