README for MultiSiteSync - Thanks AnnnA for the name =) Beats usersync.sh

DISCLAMER: Any use of the word "he" in this readme should be read as "she" if
you are female, or otherwise funky.
If this script fucks up your userbase, be happy you had that backup before starting
playing with it.

This script is not for everyone. It requires knowledge of sharing folders over the
internet in a secure way. I wont help with that part if you get stuck.

----------------------------------------------------------------------------------
Before you read any further, be adviced that this script only works if you are
using ONE stat section. You can still use the syncing feature and credit transfers 
with more, but you can not move stats back to the hub. There will simply be too
many calculations and stats sections that must match over several sites...

If you find any errors in this readme, please let me know. Its tough keeping it
updated between versions ( efnet - #glftpd - Turranius ).
----------------------------------------------------------------------------------

index. Search for <# number> to get there.
 1.1 = Questions and Answers (Q&A).
Slave part:
 2.1 = MSS-CORE.SH / MSS-SLAVE.CONF
 2.2 = TRANSFERCREDITS.SH
 2.3 = SLAVESERVER.SH
 2.4 = MSS-RLSCOMP manual
 2.5 = MSS-CUSTOM manual
Hub part:
 3.1 = MSS-HUB.CONF manual.
 3.2 = MSS-SITE / MSS-POST manuals
 3.3 = MSS-CUSTOMHUB manual
General part:
 4.1 = Actionfile in deep and personal
 4.2 = Replication system.

#############################################################################################
# 1.1 : Questions and answers
#############################################################################################

"What is this?"

These are a few scripts ment for people who wants to run multiple sites
with the same userbase.

*

"But I already do that by mounting etc and users from one site to another."

Sure, that works. But what happens when the source go down? All other sites
go down too. I didnt like that. 
Plus the fact that when the bandwidth on the source site is high, it is 
terribly slow to log on to the other site, unless you know your firewall =)

So this script allows for each site to keep its own userfiles, including passwd
and group file, but syncing it from a central point. That way you will have one
HUB server and one or more SLAVE servers. The slaves run MSS-CORE and will update its
own userfiles with those from the hub. 
How often it does this is entirely up to you and how you run it. You may crontab the script
to do a full sync as often as you like and there is also an actionsystem in place that will
lead to near realtime syncronisation.
All changes will be made on the hub server and then replicated, using this script, to 
the slave server(s).

The users wont notice if the replication is slow, because its all done in the 
background.

*

"Who made these scripts?"

The first versions were made by Turranius (myself). Since version 2.0 though,
void0 has helped with a lot of stuff, including getting us going on the actionfile
system etc etc. void0 is also responsible for mss-rlscomp.sh which will tell mss-core.sh
to move stats after each race is finished.

*

"Why so many scripts? And whats the 'hub' and 'slave' folder in the package?"

Ok, lets describe the scripts.
slave/mss-core.sh    - This is the main mss script ment for the slave(s). This one
                       does all the sync/move stats work.

slave/mss-slave.conf - This is the config file for mss-core.sh. Easier to upgrade
                       if the config is seperate.

slave/mss-slave.id   - This contains two lines. GLROOT and DESTNAME.
                       All other scripts can be updated from the hub incase of new
                       versions, but the DESTNAME setting must be unique, so that is
                       kept seperate. GLROOT can also differ so thats kept in here too.

slave/mss-rlscomp.sh - For zipscript-c. After a release is complete, this will be run
                       to write all usernames to the actionfile so the stats are moved
                       after each race. 

slave/mss-custom.sh  - This is for the actionfile system. If you want some kind of sync
                       when doing any kind of command, you can use this as a cscript.
                       More info in the script itself or see section 2.5.

slave/transfercredits.sh - Since credits are seperate for each site, this is 
                           the script used as a custom command on the slave to
                           transfer creds between hub<->slave.

slave/slaveserver.sh     - Just used as a custom command on the slave to disable
                           commands like adduser, chgrp etc, since all that should
                           be done on the hub.

Thats all thats needed for the slave.

hub/mss-site.sh    - This is a script ment as a custom command for the hub.
                     Its kinda like an admin interface where you can force
                     syncs, fetch the logs from the slaves etc etc.

hub/mss-post.sh    - Used as a cscript on the hub for all administrative commands
                     like addip, adduser, purge etc etc. It creates the actionfiles
                     that mss-core.sh reads and executes.

hub/mss-hub.conf   - Config file for mss-site.sh & mss-post.sh

hub/mss-hub.id     - Like for the slave, This contains SOURCENAME and GLROOT.

hub/mss-customhub  - Same as mss-custom.sh for the slave, but this is used for the
                     hub for triggering syncs/stats moves on any command you like.

And thats all thats needed on the hub, for now =)

*

"Ok, what will mss-core.sh sync?"

mss-core.sh will do most of the work. It will sync users/move stats etc etc.
When syncing, the following fields are checked from the hub:
GENERAL
LOGINS
FLAGS
TAGLINE
RATIO
GROUP
PRIVATE
IP

It will keep track of primary groups for a user, so it wont mix the group order
up.

As you notice, it does not sync SLOTS. Why? Well, any changes you do to users
should be made on the HUB server. That is why mss-core.sh also comes with an option
to remove flag 2 (gadmin) from all users on the slaves. It dosnt matter if they are 
gadmins there too (since we disable adduser on the slaves), but it will look better 
and make it easier for the users to understand.

*

Now then, what if a user does NOT exist on the slave (just added to hub).
In that case, it will simply copy the userfile over, setting CREDITS and all
the stats to 0 ( 0 0 0 actually. 0 0 0 0 for TIME ).
That includes:
CREDITS
ALLUP
ALLDN
WKUP
WKDN
DAYUP
DAYDN
MONTHUP
MONTHDN
NUKE
TIME
SLOTS

So, the users will have a nice and clean userfile.

*

"But what about passwd and group files?"

The first thing it does, before checking any users (hm, this readme is backwards)
is to run md5sum on both the hub server's passwg/group files, then on the slave server. 
If they do not match, it will simply overwrite the file on the slave. Well, since v2.0, it
will copy the hub's to a temp location on the slave. Then check if the md5sum still matches
the one on the hub and if it does, overwrite the slaves file.

The above functions will only READ from the source site, so no changes will be made
to the hub servers userfiles/passwd/group from the slave.

The actionfile system (mss-post.sh) will tell the slave to sync passwd and/or group
file whenever it is needed for the command executed on the hub.
For instance, adding a user called 'gurra' would result in a sync of passwd as well as 
the user himself. See section 4.1 if you dont understand the actionfile system.

*

"Stats?"

Yepp. Stats can be moved. It will move the stats from the slave and hub server
add it together, then write the added value to the hubs userfiles. 
After that, it sets all stats on the slaves userfiles, to 0.
This is so any trial scripts or similar can be run on one server (the hub).
It is the only time, part from transfering credits, that we write to the hubs
userbase.
Which stats it will move is entirely up to you. Perhaps you just want ALLUP and ALLDN
to be moved the hub? No problem. There are settings for each of them.
Perhaps you dont want stats moved? Keep it seperated if you want =)

This can either be crontabbed at predefined times to move stats from all users or 
you can use the mss-rlsconf.sh script, ment for zipscript-c, to move stats after
each race.

*

"What is this 'actionfile' system I read about?"

Previous versions only did a fullsync at crontabbed intervals. Say every 60 minutes.
This ment that if you changed or added a user on the hub, it could take up to
60 minutes before the change replicated onto the slave.
With the new actionfile system, its done within a minute. 

How?

You load mss-post.sh as a POST script on the hub, for all the administrative commands, like
adduser/chgrp/purge etc. What it does is write to a file.
Say you add an ip to the user turranius. mss-post.sh will then write to a file:
SYNC USER turranius

When mss-core.sh runs with argument "actionfile" from the slave, it will read that line
and sync user turranius. Then it will clear the actionfile.

Most commands do not execute the POST script if the command failed, ie a user tries to 
add another user, but dont have the permissions to do so, the POST script is not executed.

Not all commands are like that though. BUT, the neat thing is, if you add an ip to a user,
it dosnt actually write "Add this ip" in the actionfile. It simply says "Sync this user from
the userfile on the hub". So even if a user manages to write to the actionfile, if its not 
actually done to the hub's userfile then it wont do crap to that user.

*

"How does it get the sources userfiles?"

That part is up to you. As I said, some sites who run this setup usually just NFS
mount the hubs 'users' and 'etc' to the 'users' and 'etc' on the slave server.
Per default, I want you to mount the hubs /glftpd/ftp-data/users, to 
/glftpd/ftp-data/users.ext on the slave server.

Same with the etc folder. Mount the hubs /glftpd/etc to /glftpd/etc.ext on the slave.
I dont care how you do this. If you are using NFS and firewall, here is a nice article:
http://nfs.sourceforge.net/nfs-howto/security.html

I will not give support for how to solve this. You can do it in anyway you like, as long
as the slave server can access the userfiles/group/passwd from the hub.
It requires READ access for syncing users and READ/WRITE for moving stats.

*

"Speed?"

How long the syncing or moving of stats take is entirely up to your line. If its loaded
it will take a lot longer. MSS will read the userfiles quite a few times, so it wont be 
that fast.
It does, however, come with an option to first copy the hubs userfiles to the slave
server, and sync from those files. This is much faster and highly recomended.

The actionfile system does not copy the files locally since its usually just one user
it checks at a time. No point to copy files for that.

*

"Backup?"

MSS also come with an option to backup all and every files before starting on a full sync. 
You can also set how many backups it should keep in the backup folder before deleting the 
oldest ones.
If you also use the option to copy the hubs userfiles to the slave (see above), then
it will backup those too. So the slave(s) will hold a backup of both its own, and the hubs
userfiles. Neat Neat.

Same as above, it will not backup when just running the actionfile system, only on full,
fullstats and fullsync. mss-core.sh also have an option to just run the backup part when
executed from shell. You can run mss-core.sh on the slave, without any arguments, to get
a full list of functions.

*

"Multiple slaves?"

Yes, in theory, you can have as many slaves as you want. Just mount the hubs users
and etc folder to each one and run this script. 
If you have lots of sites linked, you can also make one or more servers act as both
slave and hub. Just make it a normal slave, then share its users/etc to another computer
who runs this script too. Updates to userfiles will obviously be slower, but it works =)

*

"Logging?"

Oh yeah. Dont be fooled by the number of lines this script has. Half of it is just logging.
There are 3 types of logging.
1: Normal log : All changes to userfiles and stats are logged here.
2: HighDebug  : It will log everything it does. Very spammy but good never the less.
                If you have a problem, this is the log I want.
3: Debug      : Normal debug. Running it with the argument 'debug' will show on screen
                what it does (not logged to file). Almost the same as HighDebug, but to
                screen instead of file. Not quite as "deep and spammy" as HighDebug.

Syncing users, moving stats and the highdebug can be 3 different logs if you like. 
Easier to read as stats-sync can get quite spammy, even without highdebug.
Logging can also be turned off entirely (maniac).

Its recomended that you keep highdebug output to a seperate file incase something goes
wrong somewhere.

*

"Permissions?"

You can select which permissions it should chmod the userfiles too. I dont know if its
just me, but glftpd has an annoying feature to change permissions on my userfiles so root
cant change in them. This, ofcourse, can be turned off.
It will also use this value when chmodding log files/actionfiles etc etc.

*

"What if the link goes down as it is syncing. Will it delete a user because he cant be read?"

No. We do a LOT of checks to see if the link is ok here. When I find a user that is on a slave
but not on the hub, I will do the following.
If its running with COPYLOCAL on TRUE, that is, copy the userfiles locally, and the slave servers
user does not exist in the copylocal folder, it will switch back and try and check from the the
hubs users folder. If he does exist there, that means the copy of userfiles were interuppted, and
he will not be deleted, just skipped for syncing this time.

If however, he does not exist as a userfile in either the slaves local copy or the hubs userfolder
it will check with the passwd file. The passwd file is already synced at this stage, so it will be
up to date. 

If he does NOT exist in the passwd file, he must be deleted, and will be deleted on the slave too.
However, if he do exist in the passwd file, that can only mean two things
1: The link went down. Cant read the userfolder on the hub.
2: The usersfolder vs passwd file are not synced on the hub. He has a passwd entry, but no
   userfile.
In either case, he will not be deleted on the slave.

MSS will do one more thing to make sure it does not delete anyone by mistake IF you use the
backup feature.
It will make a directory, if it dosnt exist, in the backup dir, called "deleted".
Then, instead of deleting the userfile, it will move it to that dir.

If it should happen to find him on the hub server again later, it will make a new clean user
of the hubs file BUT log that there is an old userfile for this user. You can then decide to 
either delete that old file or restore it over the newly created one.

If you do not use the backup feature, the user will simply be deleted.

As I said, we do a lot of checks for link or permission problems. Before every write
to a userfile, it will verify that it CAN do it first. If not, it will report it and leave the
user alone.

--------------------------------------------------------

"Ok, so whats this transfercreds.sh script?"

The transfer credits script is loaded as a custom command on the slave server(s).
It allows a user to transfer credits back and from the hub and slave. 
It does not permit moving of credits between slave -> slave. Atleast not yet.
For now, if you have more then 2 sites, and the user wishes to move creds from one
slave to the other, he will have to first move it to the hub, then log on the other
slave and grab the credits from the hub.

This script is only ment to be loaded onto the slave servers. If you wish to load
it on the hub server as well, you have to mount the users from the slave server onto
the hub too. Kinda lot of work, but it works both ways.

So if a user wishes to move 500 megs to the hub, he would type:
site transfer give 500

If he wishes to move 500 megs from the hub, he would type:
site transfer take 500.

It also have some built in aliases.
Instead of give, you can use: put, throw
Instead of take, you can use: get, fetch
Pretty easy to add your own if you just look in the script.

It comes with its own logfile, as well as a limit on the min/max limit to move.
You can also check your current credits on both stats with the argument 'status'
as well as the state of the link.

Edit it to set all the paths etc. More on that later.

* END OF Q&A.

#############################################################################################
# 2.1 MSS-CORE.SH / MSS-SLAVE.CONF
#############################################################################################

Oki, lets go over the options of the scripts. Lets do this in the order they should be installed
starting with the big one, mss-core.sh on the slave(s).

mss-core consists of 3 files. mss-core.sh, mss-slave.conf and mss-slave.id. 
Put mss-slave.conf anywhere you like. Preferebly in the same dir as mss-core.sh. If you not put
them in the same dir then you will need to edit mss-core.sh and specify where the config is.

Start editing mss-slave.id. Only 2 settings in here

GLROOT=         The path to glftpd. Default is just /glftpd

DESTNAME=       This slave servers name. Will be used in the logging and when reading
                the actionfile. Make sure this is unique for all your slaves.

Now, edit mss-slave.conf
NOTE: The hub server is called the source and the slave is called dest, short for destination.

SOURCENAME=     The hub servers name. Will be used in the logging.

USERSOURCE=	Where are the hubs source files mounted? The default is
		/glftpd/ftp-data/users.ext
                As I said before, HOW you mount this is up you you, be it samba
                NFS or a friggin laplink cable. As long as MSS can READ it for the
                syncing of users and WRITE to it if you want to move stats back 
                there from the slave(dest, this computer).

USERDEST=	Where is this slaves userfiles located? This is the "live" dir. Usually
		just /glftpd/ftp-data/users

PASSWDSOURCE=	Where is the hubs passwd and group file?
                Imagine you mounted the hubs /glftpd/etc folder to /glftpd/etc.ext on
                the slave. In that case, make it /glftpd/etc.ext

PASSWDDEST=	Where is the passwd file on this slave? Usually /glftpd/etc/passwd

BACKUPDEST=	This is where it will make backups. The default is /glftpd/ftp-data/backup
                so you have to create that folder if it dosnt exist.
                If you leave this empty, it will not make backups (not recomended).

MAXBACKUPS=100	How many backups should I keep in the above folder before I start removing
                the oldest ones? Make sure you read and understand the NOTE part above.

PINGTEST=       TRUE/FALSE. Do you want to do a ping test to the hub before running?
                If your hubs firewall accepts pings, I suggest you set this to TRUE.
                It will send 2 pings. If 2 pings are recived, it will simply continue.
                If 1 ping goes through, it will log to the HIGHDEBUGLOG if that is on and then
                continue as normal.
                If 0 pings go through, it will umount PASSWDSOURCE and PASSWDDEST and quit.
                Umounting will be done with umount -f, umount -l and finally a umount.
                Errors here are shown on screen when running with 'debug' and in the log.

PINGOPTIONS=    What options to give the ping command, if set to TRUE above. What you want it
                to do is send 2 pings only, so check 'man ping' for how to get it to do that.
                On RedHat/Mandrake, its -w2 so that is the default. According to [Acido], you
                want to use -c2 on Debian.

USENFS=         TRUE/FALSE. If you are mounting the hub shares with NFS, you can enable this
                one and it will verify, with rpcinfo, that mountd, nfs, stats and nlock is 
                answering on the hub. If one of them does not, it will unmount in the same
                way as PINGTEST does ( very hard! ).
                You may use this as well as PINGTEST if you like.
                Errors here are shown on screen when running with 'debug' and in the log.

DNSHUB=         This is used by PINGTEST and USENFS only. Basically the name of the hub.
                You can use either the DNS name or the IP. If you leave this one blank, it
                will attempt to read the name from /etc/fstab where I suppose you mount 
                USERSOURCE and PASSWDSOURCE. Having the mounts in /etc/fstab and leaving this
                blank is recomended.
                Errors here are shown on screen when running with 'debug' and in the log.

REMOUNT=        If you mounted users.ext and etc.ext so they show up when you run 'df' and you
                have the entries for them in the /etc/fstab, then you can set this to TRUE.
                What it will do is; if it can not find the VERIFYUSER (below), it will attempt
                to unmount users.ext and etc.ext and mount them again. If the hub server is on a
                dynamic IP and it changes, the slave servers will loose contact with it and
                MSS will stop running, so set this one to TRUE and you dont have to bother
                about it. Strongly suggested to use this with PINGTEST and/or USENFS since those
                umounts the drives if they find any errors, and REMOUNT mounts them again when
                the hub comes back up.

VERIFYUSER=	A username that must exist on both the hub and the slave. I will use this 
                user to do some checks. Mostly to see if the link is up. 
		The default is glftpd, because that user is there by default.

LOG=		Which file should I do all the usersync logging to? The default is
		/glftpd/ftp-data/logs/usersync.log
		You can also make this empty to disable logging (not recomended).

STATLOG=	You may want a seperate logfile for moving stats with the 'stats' argument
		since that can be quite spammy in itself. If you want it in the same log as
		above, just use the same file.
                This will only log the actual moveing of stats. Errors and others will still
                be logged to LOG above.
		As above, leave this empty to disable logging.

HIGHDEBUGLOG=   Where to log highdebug output. You can set this to a seperate file or 
                to the same as LOG or STATLOG. To disable this one, look below at
                HIGHDEBUG.

TMP=		A temporary folder where I will edit userfiles and stuff.
		The default is /glftpd/tmp

EXCLUDE=	Any users you do NOT want synced can be added here. Only users needs to be
		added since I will make sure its really a userfile I am trying to read
		(by grepping FLAGS, and making sure the size is somewhat valid).
		The default is just: EXCLUDE='default.user'
		Seperate users with a | and make sure you encase it with ' ', as the default.
		so EXCLUDE='default.user|pella|pekka|blabla'

                Note that you can not have a user that is not on the hub though. I move passwd
                and group files, so if he exists only on the slave, he wont get any password and
                cant log in.

MD5BIN=		To verify that the passwd/group files match with the hub, MSS will use
		md5sum. This must be installed, and this is the full path to it.
		Default is /usr/bin/md5sum
                To find where it is, either do 'which md5sum' or 'locate md5sum' from shell.

LOCKFILE=	Where to put the lockfile so the script does not run once its already running?
		The default is: /glftpd/tmp/usersync.lock
                Note. If the slave goes down in the middle of a sync, the lockfile will still bere
                there and the script wont run. I therefor recomend you add the following to your
                /etc/rc.local file (or whatever your system runs at bootup) on the slaves.
                if [ -r /glftpd/tmp/usersync.lock ]; then
                  rm -f /glftpd/tmp/usersync.lock
                fi
                That is, ofcourse, if that is where you set the lockfile too =).

                Note: Since version 2.0, mss tries to detect wheither its actually running or not
                      when detecting a .lock file at startup. This procedure does not seem to be 100%
                      so do the above anyway just in case.

PERMS=		What to change the permissions to on all userfiles I encounter? The default is
		777. Make this empty to disable the feature, but be prepared that MSS will complain
		that I cant read or write to certain userfiles if so.

RUNLOCAL=	TRUE/FALSE. Do you want the to first copy the userfiles from the hub to the 
		temporary folder on the slave, then sync from there? This is HIGHLY recomended
		as speeds will improve alot by it. It also has the added benefit of backing
		up a that copy on the slave (as well as the slaves usersfolder), if you
		have not disabled backups above.

NOGADMINS=	TRUE/FALSE. With this on TRUE, MSS will strip flag 2 from any users on the slave.
		Recomend we do this since they shouldnt add users on the slave(s) anyway.
                It will not do this when adding the user, but the time after when its syncing it.

SHAREALLOTMENT= TRUE/FALSE. Do you want the same allotment on the slave as you do on the hub?
                If you set this on TRUE and the user has 10gig allotment on the hub, he will get
                the same on the slave, thus doubling his credits (since he can move it between sites).
                You can halve the amount on the hub and he will get 50% of the orignal value on the
                hub and 50% on the slave.
                If, however, you like to keep weekly allotment on the hub ONLY, set this to FALSE.
                When FALSE, it will set any weekly allotment, on the slave, to 1000 Kb. That means
                the user will get 1 Mb credits on the slave when the weekly allotment resets.
                He will then have to manually move credits from the hub to the slave.
                It will not change allotment when adding the user, but the time after, when syncing.
                * This function is new in 1.1. Previous behaviour was the same as TRUE.

SLOWTIME=	Between each userfile, you have the option to wait for a while. This is to preserve
		both CPU and bandwidth, or if you are searching for an error and dont like it to
		just spam by. Set the number of seconds to wait between each userfile.
		The default is 0.1 so the HD's get a little break.
                If you get sleep errors, try 0,1 instead of 0.1
                If is not recomended to set this to 0, but hey, its your system and some distros wont
                allow commas in sleep ( non posix ).

HIGHDEBUG=	TRUE/FALSE. Do you want to use highdebug at all?
		Also note that you can execute mss-core.sh with the argument "debug" and it will do
		basically the same, but to the screen. Not everything it does is displayed in that
		mode though.

MOVEALLUP=	TRUE/FALSE. When executing mss-core.sh with the argument "fullstats", it will move all 
		stats from this slave back to the hub. This is the only time mss-core.sh will 
		write to the userfiles on the hub.
		Once the stats are moved, it will set the stats to 0 on the slave.
		Here you decide which stats should be moved. If you like to keep daily or 
		weekly stats on the slave, but move allup/alldn back to the hub, set that here.

MOVEALLDN=	See above
MOVEWKUP=	See above
MOVEWKDN=	See above
MOVEDAYUP=	See above
MOVEDAYDN	See above
MOVEMONTHUP=	See above
MOVEMONTHDN=	See above
MOVENUKE=	See above

WITHDRAWNUKE=   TRUE/FALSE. Heres the thing: When stats are moved from the slave to the hub and
                the user is nuked on the slave, he gets to keep his stats since they
                will not go below 0 on the slave. So, this one will check the NUKE field
                on the user when syncing stats. If he's nuked, it will subtract the MB amount
                from his stats on the hub.
                If you would rather have the user keep his stats, set this to FALSE and skip
                the next 4 settings.

WITHDRAWALLUP=  TRUE/FALSE. Subtract nuked stats from ALLUP? 
WITHDRAWMONTHUP=TRUE/FALSE. Subtract nuked stats from MONTHUP?
WITHDRAWWKUP=   TRUE/FALSE. Subtract nuked stats from WKUP?
WITHDRAWDAYUP=  TRUE/FALSE. Subtract nuked stats from DAYUP? This one is up to you.
                If he was upping something yesterday and you nuke him today, it will
                subtract the nuked stats from todays uploaded stats on the hub, even though
                his stats there are 0. 
                It will never go below 0 on the hub either though, so no biggie.

FETCHLOGSDIR=   This is the dir the keep actionfiles in. It should be a folder that the slave
                can reach, located on the hub. Therefor, the default is 
                $PASSWDSOURCE/msslogs since PASSWDSOURCE is mounted above. 
                ( /glftpd/etc/msslogs/ on the hub ).

*               There are also settings for mss-rlscomp.sh, mss-custom.sh and
                transfercredits.sh below. 
                Those will be covered their respective section below.

mss-core.sh can do a few other things as well. To check, configure mss-slave.conf, then 
run mss-core.sh from shell.

* The first time you run a full sync, leave the users folder on the slave totally empty so it can
* make an exact match. Any previous users in the slaves users folder will most likely be
* deleted if they are not on the hub too.
* You cant NOT have users on the slave that isnt on the hub. The slave is a ... slave only!

Thats everthing there is to know about mss-core.sh. Run it manually a few times or something, then
crontab it.

The crontab should be made to do the actionfile. I run it every minute since it just checks
if there is an actionfile. If so, sync what it says and if not, quit.
NOTE: If you crontab actionfile like the example, put this one below any 'full' or other
sync mode. Otherwise you run the risk of mss not running 'full' cause 'actionfile' starts
slightly before 'full'.
* * * * * /glftpd/bin/mss-core.sh actionfile

If you also wish to move the stats back to the hub twice a day, this line will do it at
11:50AM and once again at 11:50PM.
50 11,23 * * * /glftpd/bin/mss-core.sh fullstats
Otherwise, just use the mss-rlscomp.sh script to move stats after each race.

To be on the safe side, do a full sync instead of fullstats.
50 11,23 * * * /glftpd/bin/mss-core.sh full

NOTE: If you have more then one slave, play it safe and spread out the time so the slaves
      does not all sync stats at the same time. I try to check so this does not happen in
      the script itself, but it never hurts to be on the safe side.

#############################################################################################
# 2:2 TRANSFERCREDITS.SH
#############################################################################################

Options in transfercredits.sh

transfercredits.sh is the script that is ment to be loaded in the slaves glftpd.conf
file, allowing the user to move credits to and from the hub.
Since it can not move creds from slave <-> slave, its kinda limited. Make me a better one then!

The configuration is set in mss-slave.conf. It also reads GLROOT from mss-slave.id or if you
hardset that in the script itself (leave it in mss-slave.id if possible).

Zie options

COMMAND=	This is just the command you use for the script in glftpd.conf
		the default is "transfer"
		Its just for the help texts, so it matches what you set it to.

TRANSFERLOG=	Where to log user transfers? It will log before doing any transfers
		so if the script craps out, or the users claim they didnt get the
		creds, verify with the log.
                It will also log any shit the users try to pull. Like site take -1000 or so.
		Default is /ftp-data/log/transfers.log
		Its relative to the GLROOT, set in mss-slave.id.

TRANSFERTMP=    A temp folder MSS will use to edit userfiles.
		Default is /tmp
		Its relative to the GLROOT, set in mss-slave.id.
                This can be the same directory as TMP, but without the $GLROOT.

BIN=		The location of glftpds bin folder, relative to GLROOT.
                Had one user who this was different for, so I made it an
                options.

MINAMOUNT=	The minumum amount of megs the users can transfer at once.
		The default is "10" (Mb).

MAXAMOUNT=	The maximum amount of megs the users can transfer at once.
		The default is "10000" (10Gb).

AUTOMATICSTATUS= TRUE/FALSE. When someone transfers credits, the previous and the
                 new credits will be displayed at the end so the user can verify
                 that the transaction took place ok. Its like doing a 
                 'site transfer status' before and after moving credits.
                 This might take a second or so longer, so its an option too.

Naturally, it uses parts on the general config too, like SOURCENAME and the paths to
userfiles.

Thats all about transfercreds.sh. Load it on the slaves glftpd.conf, like
site_cmd TRANSFER       EXEC    /bin/transfercredits.sh
custom-transfer         *

Running 'site transfer' will give you the options. There's not that many.
It will probably spit out some error about missing files, but it should be 
fairy obvious what to do.

If you find any exploits in this one, please let me know. I've done
everything I could think of to make it as secure as possible.

Oh yeah. glftpd or something has a nasty way of chmodding userfiles so users get access
denied on them. The lazy way to solve this is by adding this to crontab:
* * * * *     chmod 777 /glftpd/ftp-data/users/* >/dev/null 2>&1


#############################################################################################
# 2.3 SLAVESERVER.SH
#############################################################################################

slaveserver.sh options

This really just outputs some text. Weither you want to use this is 
up to you. Open it up and change:

SOURCENAME=	The name of the HUB. Should match SOURCENAME in mss-slave.conf but dosnt have to.

Changes the text as you please. I disable the following commands by
writing this to the slaves glftpd.conf since those commands should be executed
from the hub or they will be overwritten on a full sync.

site_cmd TAGLINE        EXEC    /bin/slaveserver.sh
site_cmd ADDUSER        EXEC    /bin/slaveserver.sh
site_cmd CHANGE         EXEC    /bin/slaveserver.sh
site_cmd CHPASS         EXEC    /bin/slaveserver.sh
site_cmd ADDIP          EXEC    /bin/slaveserver.sh
site_cmd DELIP          EXEC    /bin/slaveserver.sh
site_cmd GADDUSER       EXEC    /bin/slaveserver.sh
site_cmd PURGE          EXEC    /bin/slaveserver.sh
site_cmd GRPNFO         EXEC    /bin/slaveserver.sh
site_cmd GRPREN         EXEC    /bin/slaveserver.sh
site_cmd GRPADD         EXEC    /bin/slaveserver.sh
site_cmd GRPDEL         EXEC    /bin/slaveserver.sh
site_cmd FLAGS          EXEC    /bin/slaveserver.sh
site_cmd CHGRP          EXEC    /bin/slaveserver.sh

custom-tagline          *
custom-adduser          *
custom-change           *
custom-chpass           *
custom-addip            *
custom-delip            *
custom-gadduser         *
custom-purge            *
custom-grpnfo           *
custom-grpren           *
custom-grpadd           *
custom-grpdel           *
custom-flags            *
custom-chgrp            *


#############################################################################################
# 2.4 MSS-RLSCOMP manual
#############################################################################################

mss-rlscomp.sh goes to the slave. Its the script ment for zipscript-c. Install it by editing
zsconfig.h and set:

#define enable_complete_script TRUE
#define complete_script        "/bin/mss-rlscomp.sh"

Note, that is not a script so the #'s should be there. Dont remove them.
Once added, recompile zipscript-c as usual.
( make clean; ./configure; make; make install )

Make sure mss-rlscomp.sh is executable by everyone (755 or so).

This means that when zipscript-c completes a release, it will run mss-rlscomp.sh with the
last file as argument. mss-rlscomp will check which directory this file is in, by reading
xferlog, take all usernames and write it to a file in your etc folder on this slave.
This file will then be read by mss-core.sh when executed with 'actionfile' and move stats
from those users that were included in the race.

mss-rlscomp.sh takes its settings from mss-slave.conf and mss-slave.id.

The settings in mss-slave.conf for mss.rlscomp.sh:

 GLXFERLOG="$GLROOT/ftp-data/logs/xferlog"           This is your path to the xferlog file.
                                                     As read above, GLROOT is taken from mss-rlscomp.sh
                                                     and should be kept as $GLROOT here.

 EXCLUDE="/site/0day|/site/mp3"                      If you have sections where it should NOT move stats when
                                                     a release is complete, add them here, | separated.
                                                     Start from /site here. Not from /glftpd.
                                                     If your affils do not get stats until they pre, you should
                                                     add your predir here too. Like /GROUPS/

 RLSCOMPLOG="$GLROOT/ftp-data/logs/mssrlscomp.log"   This is where mss-rlscomp will log to. Create the file and 
                                                     set something like chmod 766 on it.

 VERBOSE="FALSE"                                     TRUE/FALSE. When testing from shell, you should set this to
                                                     TRUE. When running it live from zipscript-c, you MUST set it 
                                                     to FALSE. <- REMEMBER THAT! =)

Since this is executed from inside glftpd, make sure everyone have read rights on all
involved directories, including /glftpd/etc and /glftpd/etc/passwd. Everyone must also have
write permissions to the logs dir and the RLSCOMPLOG file.

#############################################################################################
# 2.5 MSS-CUSTOM manual
#############################################################################################

This is basically a script ment to be used as a pre/post script to any other command. 

Make sure your actionfile system works before playing with this one. See section 4.1 below
for more information on how the actionfile system works.

What it writes to the actionfile can be defined with the SEND setting in the script.

For instance: You want it to sync full stats when someone does a pre? Add this to glftpd.conf
 cscript  site[:space:]pre post /bin/mss-custom.sh
Then set SEND to 'SYNC STATS *'

That will send SYNC STATS * to the actionfile whenever one does a site pre.

*-[ IMPORTANT ]-************************************************
You can use it as a post command, but that does not appear to be
working on custom scripts, only glftpd internal commands, or so.
I havent had time to check which post works on and which dosnt.
*-[ IMPORTANT ]-************************************************

You can use it as a pre command as well.
$2 can be used in SEND for the username of the person giving the command and $3 can 
be given for the users primary group (although no functions exist for groups in mss-core.sh).
Use a * to execute it on all users.

Commands you can use in SEND, accepted by mss-core.sh.
SYNC STATS <user/*>   = Move stats back to hub.
SYNC BASIC <user/*>   = Sync userinfo.
SYNC FULL <user/*>    = Sync userinfo and move stats.
FETCHLOG              = Move this slaves logs back to hub.
INTEGRITY             = Check integrity of userfiles/passwd
DELREPORTS            = Remove all reports on the hub from this slave.
DELLOGS               = Delete all logs on the hub from this slave.
BACKUP                = Run a userbackup.

If you want another function for another command, just copy the script to another name, set 
in SEND what it should do and add a cscript in glftpd.conf, with the command that you want
it to trigger on.

Logging from this script will go to the same log as mss-rlscomp.sh uses. 
You set that in mss-slave.conf.

mss-core.sh will complain to the LOG (defined in mss-slave.conf) if you set commands here 
that is not accepted by mss-core.sh.


And that concludes the installation on the slave. Lets move over to
the hub installation and the actionfile system.

#############################################################################################
# 3.1 MSS-HUB.CONF manual.
#############################################################################################

Scripts for the hub server.
mss-site.sh
mss-post.sh
mss-hub.conf
mss-hub.id

Copy all 4 files to /glftpd/bin.

Start editing mss-hub.id. Set the hub's name and path to /glftpd.

Make mss-site.sh and mss-post.sh executable (755 or so).
mss-site.sh and mss-post.sh reads its settings from mss-hub.conf and mss-hub.id.
The reason a few settings are in mss-hub.id instead in the scripts themselfs is because
you can have multiple hubs. If so you might want to use the replication system to move
out new version of hub userfiles. With the settings as they are, you dont have to edit
anything in mss-site.sh and mss-post.sh but can instead just push them out.

Lets move over to config mss-hub.conf. That is used for both mss-site and mss-post.
It is described in the conf as well, but lets go over it anyway.

SLAVES=         Names of the slaves to write actionfiles to, space separated.
                These names must match the name in DESTNAME in mss-slave.id as
                configured on the slaves. So if you have 2 slaves and they are
                DESTNAME="Slave1" on the first and DESTNAME="Slave2" on the other, make
                this one; SLAVES="Slave1 Slave2"
                
GLETC=          Default: "$GLROOT/etc"
                The path the the /glftpd/etc path on the hub. GLROOT is defined in the
                mss-post and mss-site scripts. It has to be done that way since they can
                be executed from both shell as well as from inside glftpd and chrooted
                from shell...

GLUSERS=        Default: "$GLROOT/ftp-data/users"
                The path to your users folder on the hub. As above, use $GLROOT to define
                the path to /glftpd.

LOG=            Default: "$GLROOT/ftp-data/logs/msspost.log"
                Where to log to? Both logs from mss-post and mss-site goes here.

MSSLOGS=        Default: "$GLETC/msslogs"
                This should match the folder defined as FETCHLOGSDIR in mss-slave.conf
                on the slave, but seen from the hub.
                Since FETCHLOGSDIR are set as $PASSWDSOURCE/msslogs on the slave, it will 
                be $GLETC/msslogs, seen from the hub.

MSSREPL=        Default: "$GLETC/mssrepl"
                This should match the folder defined as MSSREPL in mss-slave.conf
                on the slave, but seen from the hub.
                This folder will be used for updating/replicating files (see section 4.2).

VERBOSE=        Default: "FALSE"
                If you want to test mss-post from shell, set this on TRUE.
                Since a POST script can not put anything on screen from glftpd,
                it is designed to not say anything incase of errors (since you wont
                see it inside glftpd anyway). While this is on TRUE, mss-post will NOT
                work from inside glftpd.
                mss-site.sh works from both shell and glftpd without this setting so this
                one only affects mss.post.sh.

Ok, that covers the config.

To install the scripts mss-site.sh and mss-post.sh, edit them and they include that
information.

#############################################################################################
# 3.2 MSS-SITE / MSS-POST manual
#############################################################################################

Ok, these files go on the hub.

Start playing with mss-site.sh since it can and will display any errors. mss-post can not due
to it being a POST script but they work in pretty much the same way. If mss-site works, mss-post
should too.
Unless you changed the trigger, do a 'site mss' from glftpd and it will display a menu
for you to play with. Try a few commands and make sure it is written to the actionfile.

Both mss-post and mss-site will create one actionfile for each slave defined in mss-hub.conf.
The files, unless changed, are stored in /glftpd/etc/ on the hub and called <slavename>.actions

Make sure something is written in them for each command when you test the scripts.

mss-core.sh will, when executed from the slave with argument 'actionfile', pick up its own file and do
everything in it. Once finished, the file will be deleted.

Now that you got mss-site.sh to write stuff in the actionfile(s), install mss-post.sh if
you didnt already. Try adding a user or changing someones group etc and check that it writes
to the actionfile(s) like mss-site does.
Note, if you do 'adduser' for instance, it will write:
SYNC passwd
SYNC USER <username>

mss-core.sh will, when executed with 'actionfile', read the file and first sync passwd 
then the user.
mss-core.sh is smart when reading. It will read the actionfile first, then structure what to
do and finally do it. So even if the actionfile says
SYNC USER <username>
SYNC passwd
it will always do the passwd part first because certain commands have higher priority then 
others. Rename for instance, have the highest priority. In this case we check for deleted users
by verifying with passwd, so that one must come first.

It will also remove duplicates when reading it, so dont worry if your actionfile contains
SYNC USER turranius
SYNC passwd
SYNC USER turranius
then it will only sync turranius once. No reason to do it twice.

#############################################################################################
# 3.3 MSS-CUSTOMHUB manual
#############################################################################################

This is the same type of script as mss-custom, but for the hub.
It is ment to be used as a pre/post script to any other command. 

Make sure your actionfile system works before playing with this one. See section 4.1 below
for more information on how the actionfile system works.

What it writes to the actionfile can be defined with the SEND setting in the script.

For instance: You want it to sync the user when he issues a 'site vacation'? Add this to glftpd.conf
 cscript  site[:space:]vacation pre /bin/mss-customhub.sh
Then set SEND to 'SYNC BASIC $2'

That will send 'SYNC BASIC <username>' to the actionfile whenever one does a site vacation.

*-[ IMPORTANT ]-********************************************************
You can use it as a post command as well, but that does not appear to be
working on custom scripts, only glftpd internal commands, or so.
I havent had time to check which post works on and which dosnt.
*-[ IMPORTANT ]-********************************************************

$2 can be used in SEND for the username of the person issuing the command.
Use a * to execute it on all users.

Commands you can use in SEND, accepted by mss-core.sh.
SYNC STATS <user/*>   = Move stats back to hub.
SYNC BASIC <user/*>   = Sync userinfo.
SYNC FULL <user/*>    = Sync userinfo and move stats.

If you want another function for another command, just copy the script to another name, set 
in SEND what it should do and add a cscript in glftpd.conf, with the command that you want
it to trigger on.

Logging from this script will go to the same log as mss-post.sh uses. 
You set that in mss-hub.conf (LOG option).

mss-core.sh will complain to the LOG (defined in mss-slave.conf on the slave ) if you set
commands here that is not accepted by mss-core.sh.


-----------------------------------------------------------------------------------------
If you get confused by all this, join the club and do one step at a time.
Install mss-core.sh on the slave first. Make sure it can sync stats/users manually
from shell by executing mss-core.sh.
Once that part works, install mss-site.sh on the hub. Once that works, install
mss-post.sh on the hub, etc.


#############################################################################################
# 4.1  Actionfile in deep and personal
#############################################################################################

Here is how the actionfile system works.
The actionfile is processed by mss-core.sh on the slave. To start the procedure, you run
'mss-core.sh actionfile <debug>'.

The actionfile will contain a number of entries writtes by the other scripts
(mss-post.sh (hub) / mss-site.sh (hub) / mss-rlscomp.sh (slave).

mss-post.sh for instance, is executed as a POST script on the hub when executing commands that
deals with editing users (adduser/deluser/purge/change/addip etc). If I add a user on the 
hub, the text 'SYNC PASSWD' & 'SYNC USER <username>' will be written to the actionfile.
If I change group on a user, it will be 'SYNC GROUP' & 'SYNC USER <username>'.

It writes this file to the hubs /glftpd/etc folder. It will be named <slavename>.actions.
mss-post and mss-site contains a list of slaves you have. One file per slave will be created.

The slave have the hubs /glftpd/etc mounted as /glftpd/etc.ext and when mss-core.sh sees an
actionfile with its name on it, it will start to process it. It does this in a rather clever way
by reading through the file and building a todo list. In the above case of adding a user it
will first sync the passwd file, then sync the user. Each command has a special priority so mss-core
will run them in a specific order.

So, if we first add the user test1 and then add test2 (2 users), the actionfile will contain
SYNC passwd
SYNC USER test1
SYNC passwd 
SYNC user test2

mss-core will read the actionfile, building the todo list. It will see that SYNC passwd is already
in the queue and skip any duplicates of it. So in this case it will 
first sync passwd
then sync the userfile for test1 and finally the userfile for test2.

Likewise, if you have 2 action for the same user. Sync user and move stats, mss-core.sh will
ignore those and do a full sync on him instead. All to save time.

Renaming a user is a bit tricky, but it works and has the highest priority so it does work
even if you add a new user right afterwards with the same name. void0 did a great job on this
part ( he trashed my version of it =(( ).

When mss-core starts to read an actionfile it will rename it to .locked so that if mss-core.sh
should run again it will detect that the .locked file exists and exit. Once it is finished
processing the .locked file, it will delete it. Should a .locked file exist but it can not find
mss running as another process, it will rename the .locked to .notdone (see below) so it can
pick it up the next time it runs. That instance will still quit after renaming it though.

mss-core does a lot of checks to make sure the link is still up. Should the link go down in the
middle of a execution, it will save the actionfile to .notdone in the /glftpd/etc folder on the
SLAVE. 
The next time it runs and the link is up again, it will start with the .notdone file before processing
any new actionfiles.

This is where mss-rlscomp comes in (zipscript-c's instant stats mover). Since this is executed on the
slave, its not valid to save it to the actionfile that is located on the hub. If we did and the slave
gets a race when the hub is down, mss-rlscomp would not be able to write the actionfile and those stats
would not be moved until you did a full stat move from all users.

SO.. mss-rlscomp writes directly to a .notdone file on the slave. As we said before, mss-core will, when
running with 'actionfile', do this .notdone file first.

In conclusion, if we crontab mss-core.sh to run every minute with the argument 'actionfile', there will
be a total waiting time of max 1 minute before the user is synced or the stats from the race on the slave
are moved back to the hub.

#############################################################################################
# 4.2  Replication system.
#############################################################################################

Since version 2.1, there is a replication system in place to push out new versions to the
slave(s) from the hub. This works the following way:

mss-site.sh will, the first time it is executed from the hub (site mss), create a folder. 
By default, this is in /glftpd/etc/mssrepl on the hub but it is definable in mss-hub.conf
It has to be a folder that the slave has access to, so I used the etc folder by default.

If you, from the hub, inside glftpd, do 'site mss repllist', it will list the contents of this folder.
If will ofcourse be empty unless you put something in there.

If you get a new version of say, mss-rlscomp.sh and you are going to push that out to 10 slaves, do this:
Put the new script in /glftpd/etc/mssrepl on the hub. Then, from the hub, inside glftpd, do:
site mss replicate mss-rlscomp.sh -all
That will cause mss-site.sh to write that to the actionfile. The next time the slave(s) run 'mss-core.sh actionfile'
it will copy that file to the same location as mss-core.sh is located (/glftpd/bin).

You may also specify 'site mss replicate mss-rlscomp.sh -all /glftpd/bin', but seeing how glftpds /bin dir can be
in different locations on the many slaves, giving no 'destinationdir' argument will copy the file to the same folder
as mss-core.sh is in, on the slave. Each slave will detect where it is running mss-core.sh from.

You can also define the slavename instead of using -all if you dont want it on all slaves. Slaves are defined in
mss-hub.conf.

If the file should already exist in the slave directory it will create a folder called mss_oldfiles, copy the old
file in there and finally overwrite the old one with the new one. Restoring from mss_oldfiles must be done manually
if you (or I, tihi) screw something up.

Once updated, mss-core.sh will write a report to the FETCHLOGSDIR defined in mss-slave.conf. So, depending on how
often you crontab 'mss-core.sh actionfile' on the slave, you can check on the hub that the slaves took the new version
by doing 'site mss readreports' from inside glftpd. That is the same way you do it as if you issued the
'check integrity' mode in mss-site.sh.
If mss-core.sh finds the version number at the 2 top lines of the new file, it will report that too. ( version number
should be in the format VER= at the first or second line at the file for it to find it. )

You can replicate mss-core.sh, even though that is the script that does the updating, but the updated file will not
be used until the next time mss-core.sh runs.

Also note, you can replicate any file you wish to the slave. As long as the destinationdir exists (if entered) and
the user running mss-core.sh has write permissions there. You really should use root for this.

The backupdir 'mss_oldfiles' is always in the same folder as mss-core.sh is in, even if you specify another location
to copy the files to.

Ok, so say you want to replicate a new mss-slave.conf? No problemo. The only settings that should never be replicated
from a central point is GLROOT and DESTNAME since that can differ between slaves. Those 2 settings are located in
mss-slave.id on the slave and ofcourse, that file should NEVER be replicated (unless you only have one slave).

Other then that, every file ment for the slave can be replicated out from the hub.

IMPORTANT: Make bloody well sure the permissions on the file are ok before pushing it out to 
the slaves. If you push out mss-core.sh with non executable permissions, the slave wont be 
able to execute it.

#############################################################################################

Thats about it. Have fun =)

/Turranius and void0
