30 days data loss

Hi Guys,

Been through all the posts here and I agree with each one of you.

Here is the current state of our infrastructure :

1. Joomla is 3.x latest without user database.
2. membership and ACL maintained by amember script.
3. Forum has its own DB but depends on amember for users and permissions.
4. Kayako has its own DB but depends on amember for users and permissions.

All except Joomla are due for upgrades but it is not that easy, we have tried few times last year and we had issues. The most important of all is our forums as it has too much of user generated content. We are moving it to other script soon, so next time on we should not be losing any of forum requests. We lost over 1200+ posts in Teline V forum, the whole forum is empty and its too much of pain to see that happen.

What happened post server and DB crash is human error which was not identified before the server crash and none of us are happy about it, to see ourselves in this position again and again is sheer helplessness. Hung Dinh (CEO) and me believe in mentoring people in the company and give them all the resources and freedom to execute the responsibilities that comes with the position, while not blaming the person in charge but the human error has us both worried.

Yes, we are supposed to be having daily backups and we did have them, such is the case that they are rendered useless, I can point fingers but that does not solve the issue and we are fixing the way we operate, we never had such issues before maximum data loss in cases was limited to a day. To be frank, to be caught in such situations where we have to work our ass off to fix things which are not supposed to happen is sheer hell.

Guys in previous reply said about lunar year, yes some of our dev team is based out of south east asia where the lunar year is important but never has it happened in any year that more than 30% of staff is off, we are still in functional mode and attending support requests as routine, post this crash issue we have removed the limited support notice and the team has been asked to be online from home and work to make sure we do not have overdues. The notice of limited support is for the issues which may need us to pull back the devs who are not associated with support tasks.

I myself is handling 90% of all issues related to accounts and logins personally and not leaving it to my team as I want to be sure that all are taken care in the shortest possible time with full satisfaction of the member.

We just merged more of the missing members database today but had to force change usernames for some, earlier just 60+ users username were forced changed I do not have the exact number of forced username change for second merge as on now but its 100-200+ I think. We would be sending emails to those members for resetting their passwords and get going with the assigned username with option to get back to us to get it changed to original usernames, which would need to be done manually.

I am devastated with the loss of so much of forum data, memberships we will restore and manage but forum data was precious.

Arvind