Occasionally we hear about government officials that are supposed to be watching out for our good and instead look out for their own. When this happens, people will sometimes ask, "Who was watching the watchers." I was reminded of this last week when our Oracle Enterprise Manager Grid Control server lost network connectivity and we were not notified. A very large thunderstorm passed through and we also found out the next morning that we had switched to generator power at least three separate times during the night. Sometime during one of those switch overs, the network interface lost contact.
Of course we did not realize it was down for about 14 hours. The company uses HP's Network Node Manager which raises alerts to the on-call help desk person, but for some reason, no call was made to the system administrator that the server was unavailable. Not only do we use Grid Control to monitor all of our Oracle and People Soft systems, but we also use it as a centralized job scheduler. Once the server was available again, I figured out that we missed about 150 jobs. No backups, no disaster recovery log maintenance, no nightly application maintenance jobs, no nothing for fourteen hours. The better part of the next day was spent running backups and getting our Data Guard instances caught up since the archivelog destinations filled up.
So now what? We considered writing a shell script and using cron on one of the production servers to ping the Grid Control server and e-mail if it was not available. Obviously, this is a bit "patchwork" but should at least tell us if the server is unavailable. Another option would be to set up Network Node Manager to page the system administrator directly if a server goes down. This would be my choice, and I plan to work to get this implemented, but at this time, I don't have access to that system. In the meantime we decided to use our test Grid Control system (yes, we have one) to monitor the production Grid Control server. Of course this means a separate Agent installed on the production server, but at least we will have visability into the system should something happen agin.
The question through all of this is how much is enough? I guess you could ask the same question about watching our government representatives... although I think they need to be watched even closer!
Wednesday, May 20, 2009
Wednesday, May 6, 2009
Collaborate 09 - Wednesday
I wasn't able to write an update yesterday, but not because there wasn't anything to write about. On the contrary, I was so busy that I didn't have an opportunity to sit down long enough to write. The week has been very productive so far and will continue to be. Of course, everyone is looking forward to the party tonight and Universal Studios Islands of Adventure.
A few things that I didn't cover last time. Sunday evening we had the IOUG SIG Welcome Reception. This was very well attended and was a great opportunity to meet some new people and reconnect with people I haven't seen in a year. The Special Interest Groups each had a table and many people were interested in what the SIGs are doing. I was talking to many people about the Linux SIG and we should get some new blood into the group. Many thanks to SAP for sponsoring this event. We are always looking for vendors and friends that are willing to sponsor these events for us and SAP always seems to come through.
As one of the new IOUG Board of Directors members this year, I had my orientation yesterday to learn some more about my new responsibilities. I'm looking forward to the challenges and exciting times over the next two years of my term. My role on the Board will be Director of RUGs and SIGs. Joining these two groups under one Director makes a lot of sense. We should be able to jointly advertise for both groups as well as draw on each other's experiences and resources. Between the RUG Leader meeting yesterday and the SIG Council meeting this morning, I can already tell there is a lot of energy and great ideas from both communities.
The technical content this year is as good or better than it ever has been. All the new features of products and the new stuff people are doing with old things have been very interesting. Of course I have also been immersed in SIG meetings and Board of Directors meetings. I'm now in the last session of the day and learning about recovery scripts. An important topic, no doubt. It is very hard to believe that tomorrow is the last day already. I'm looking forward to a great time tonight with new and old friends. My presentation is finally tomorrow at 11:00. I'll be talking about the PeopleSoft Management Pack for Grid Control. I'm the very last session of the conference so I have a few giveaways to try to lure people in. Actually, I'm hoping that will get me some good reviews on my surveys!
See you on the roller coasters!
A few things that I didn't cover last time. Sunday evening we had the IOUG SIG Welcome Reception. This was very well attended and was a great opportunity to meet some new people and reconnect with people I haven't seen in a year. The Special Interest Groups each had a table and many people were interested in what the SIGs are doing. I was talking to many people about the Linux SIG and we should get some new blood into the group. Many thanks to SAP for sponsoring this event. We are always looking for vendors and friends that are willing to sponsor these events for us and SAP always seems to come through.
As one of the new IOUG Board of Directors members this year, I had my orientation yesterday to learn some more about my new responsibilities. I'm looking forward to the challenges and exciting times over the next two years of my term. My role on the Board will be Director of RUGs and SIGs. Joining these two groups under one Director makes a lot of sense. We should be able to jointly advertise for both groups as well as draw on each other's experiences and resources. Between the RUG Leader meeting yesterday and the SIG Council meeting this morning, I can already tell there is a lot of energy and great ideas from both communities.
The technical content this year is as good or better than it ever has been. All the new features of products and the new stuff people are doing with old things have been very interesting. Of course I have also been immersed in SIG meetings and Board of Directors meetings. I'm now in the last session of the day and learning about recovery scripts. An important topic, no doubt. It is very hard to believe that tomorrow is the last day already. I'm looking forward to a great time tonight with new and old friends. My presentation is finally tomorrow at 11:00. I'll be talking about the PeopleSoft Management Pack for Grid Control. I'm the very last session of the conference so I have a few giveaways to try to lure people in. Actually, I'm hoping that will get me some good reviews on my surveys!
See you on the roller coasters!
Monday, May 4, 2009
Collaborate 09 - Day 1: Charles Phillips
What a great start to Collaborate already! Charles Phillips, President of Oracle, had our keynote presentation this morning. Some highlights from the talk include some new releases, extension of support and a demo of Beehive. Charles mentioned that PeopleSoft 9.1 should be available around OpenWorld time. Not much more on that because the big deal was the announcement of EBuiness Suite 12.1. Along with that comes 9 new products that have been updated to 12.1 and 14 production that can run on 11i10 with no upgrade required. One of the main threads that came through was Oracle's sensitivity to the economic times and that not everyone can afford to upgrade. Along those lines he also announced extended support for several products without any additional cost. These included 10gR2 database, which was extended to July 2011, PeopleSoft 8.9, which was extended to Jun 2011, and EBusiness 11.9. I neglected to get that date, but they basically added a year. There were a couple other products that were extended, so I guess it's time to check out the support grid again.
Much of the discussion covered integration and collaboration. The Application Integration Model (AIA) has been expanded to include more applications and uses Oracle Middleware and web services to integrate applications. Oracle is selling the Process Integration Packs (PIP) in order to automatically integrate the applications. These are used to let Oracle worry about your integrations. Sounds great... I'd like to see it in action.
Speaking of seeing in action, there was a demo of Beehive. For many of us, we have heard the term, but are not sure what Beehive is or does. Collaboration and projects are becoming a big focus and this helps to address those issues. The big announcement today for Beehive is the release of the Team Collaboration module that goes GA tomorrow. This allows project teams to create Wikis, team calendars, versioning and check out of files all through one server. Beehive alos includes chat and e-mail and can integrate with other e-mail solutions. The demo was pretty cool and Charles said that Oracle is running this with ten to fifteen thousand users being hosted on one server.
One interesting side comment was about Exadata. Charles was talking about Exadata and how it currently uses Linux. Then he said that the Sun acquisition will not have any effect on Exadata, but that there will probably be a Solaris version at some point... interesting, very interesting.
I am now in the Darwin Awards for IT security. Some of these examples are a little funny and even more scary. I just hope never to be a recipient! More about Collaborate later.
Much of the discussion covered integration and collaboration. The Application Integration Model (AIA) has been expanded to include more applications and uses Oracle Middleware and web services to integrate applications. Oracle is selling the Process Integration Packs (PIP) in order to automatically integrate the applications. These are used to let Oracle worry about your integrations. Sounds great... I'd like to see it in action.
Speaking of seeing in action, there was a demo of Beehive. For many of us, we have heard the term, but are not sure what Beehive is or does. Collaboration and projects are becoming a big focus and this helps to address those issues. The big announcement today for Beehive is the release of the Team Collaboration module that goes GA tomorrow. This allows project teams to create Wikis, team calendars, versioning and check out of files all through one server. Beehive alos includes chat and e-mail and can integrate with other e-mail solutions. The demo was pretty cool and Charles said that Oracle is running this with ten to fifteen thousand users being hosted on one server.
One interesting side comment was about Exadata. Charles was talking about Exadata and how it currently uses Linux. Then he said that the Sun acquisition will not have any effect on Exadata, but that there will probably be a Solaris version at some point... interesting, very interesting.
I am now in the Darwin Awards for IT security. Some of these examples are a little funny and even more scary. I just hope never to be a recipient! More about Collaborate later.
Subscribe to:
Posts (Atom)
