Three Recoveries, Two Refreshes, One Delete

I feel like it's been a month since my last entry. Oh yeah, that's because it has been. I continue to be amazed with how fast time flies the older I get. Kids are supposed to make you feel younger and as they grow up, I just feel older!

The title of this post describes a full day I had a few weeks ago with a vendor application. Another title could have been "Support That Cannot Be Reached is No Support At All." The application in question has a relatively small database, but a couple hundred users. We actually converted to this centralized solution for the application about six months ago and the business owner has been converting locations two or three per week since.

I came into work to a my voice mail light... never a good sign. Apparently, they attempted to convert a large site the night before and had out grown the tablespace. "No biggie," I said and proceeded to increase the amount of allocated space. "We still can't get into the application and we cannot access any of the sites." This was very peculiar because this application has a different schema for each site and the only common information is stored in a "root" schema that manages some information on which databases are available. The user attempted to remove the affected schema from the application and I dropped it from the database, but this still did not allow the users into the application.

I'm not the most experienced person in the world, but I have dealt with quite a few applications during my career. I have never seen an application that just stopped working because of running out of space and especially not when more space was allocated. We contacted the vendor's support only to find out that their most experienced support analyst was out of town at a conference. The person we did get on the phone helped us check several things in the root schema, but couldn't seem to find anything that looked out of sync. The vendor suggested that we recover the database to prior to the load of the new schema. Knowing that the recovery wouldn't be a problem, I did have to warn them that since they had loaded the schema over twelve hours prior. This of course brings up other issues about notifications, etc, but that is a discussion for another blog posting.

Once we made the decision to go ahead and do the recovery, I looked at the dba_users view to find out when the affected schema had been created. Being the good DBA that I am, I copied the current data files to a backup location and started my RMAN recovery. Everything went perfectly and, as expected, the database came back up without the new schema. I called the application owner and told him to check it out. "It still doesn't work..." CRAP!!!!

This is the point when I start to wonder what is really going on here. It is also the point when the owner mentions that he actually loaded two locations the night before, not just the one. So, that would be recovery number two. I again figured out when the original schema was loaded and recovered to that earlier time. This time, when the owner tested the application, it worked. Okay, so at least we know which location caused the issue.

Users are allowed back into the application at this point and we plan for moving forward. The application owner says that he had tested that site in the test environment, but that he would do it again. When he tested it the second time, he found an error with the source data. Once the data was fixed, the load into test was successful. Somehow, I was talked into trying the load into production again since it worked in the test environment. I should have known better because as soon as he reloaded the site production was broken again. And that would be the explanation for recovery number three...

After the final recovery, we went ahead and copied the most current production data into the test environment. That would be refresh number one and when the data was loaded into the refreshed test environment, it broke it. At least we now had somewhere to try to figure out the issues. One more refresh and we waited to hear from the support person that new the most about the application. This is the part of the whole story that frustrated me the most. Once we had the right support person available, he was able to look and see that there was a data issue with one of the tables in the root schema. Deleted one row after loading the bad data and everything was fine. They deleted the row from the source data, reloaded into test, and everything worked. Being gun shy, we waited until the weekend to load the data into production just in case, but everything worked as expected, and he was able to load both sites without issue.

I learned a couple lessons throughout this ordeal. First, I had a gut feel that there was a data issue and not the tablespace issue that caused the application to not function. Unfortunately, I couldn't find any way to prove that and therefore had to go with the recovery that was being suggested by the vendor. This also leads to making sure the vendor has people available when needed to support issues. Of course, I didn't have anything to do about this because the vendor was chosen by the business. The other "stick to your guns" moment I had was when they wanted to load the data into production again after we thought it had been fixed. I should have insisted on a test refresh at that time to make sure we wouldn't break production again.

On the other hand, there were several good things that came out of this. My recoveries worked pretty well and I proved that I could go back to a point in time if necessary. The even better part of this was that the original data load was before the backup so we had to go back two nights and roll forward. I used to be a scripted backup and recovery person that could take or leave RMAN. After all these recoveries, to multiple points in time, I am an RMAN fan.

Well, I hope everyone had a great Christmas season, or whatever holiday you celebrated. Have a safe, happy, and prosperous new year. More posts soon. I really hope to get to this more often than I have recently. Also, make sure to look for information on Collaborate 10! See you in Vegas!

My First Two Weeks as an #OracleACE

Life has been a little interesting for me personally for the last couple of weeks and by interesting I mean not necessarily in a good way. Things are much better now, but let's just say that rushing down the hospital hall with your wife on her way to emergency surgery is something I do not want to have to experience again... EVER! She is doing well now and is on the mend, but it was a very scary experience.

Almost lost is all of that was my being named an Oracle ACE! I am very honored to be added to this incredible Oracle community. I have met and gotten to know many from the ACE community over the last couple of years and already feel very welcomed. Thank you to Michelle Malcher, Oracle ACE and fellow IOUG Board of Directors member, for nominating me for the ACE award. Thank you also to George Trujillo. George is a former IOUG Board member and Oracle ACE. He encouraged me to get started on the Enterprise Best Practices SIG when it was in its early stages. I was looking for a way to get involved and George helped make it happen for me.

Oracle has recognized the importance of Users Groups and has an entire group dedicated to supporting us. I have found that I have become a better technologist and have quicker answers to problems since joining and becoming involved with the IOUG. I am looking forward to many more years in the Oracle community and look forward to continuing to contribute as a member of the ACE community.

Interview Questions for Collaborate #c10

After much deliberation and thought, I finally decided to go a completely different direction for Collaborate 10 and do something for Professional Development. In the past I have presented on Clusterware, Data Guard, Oracle Portal, and, most recently, the PeopleSoft Management Pack for Grid Control. So, now for something completely different, I thought it would be fun to do something on interviewing.

Over the years, I have probably been on more than my fair share of interviews. Fortunately, so far, all have been by choice. The thing that always strikes me is the different types of questions that are asked. Some really make you think about your own experience and can trigger some good discussion. Others almost make you laugh. Not that I think it is a good thing to laugh at an interviewer, but at the same time they really make you wonder if this is a place you want to work. I have gathered several of these and have put together an abstract of the best and worst questions to ask or be asked in an interview. Some are down right funny and some should be standard because they show how people think and what their real experiences are.

If you want to hear what I think are good or bad, you'll have to attend Collaborate and check it out. What I would like are other examples from people that might be good to put into the presentation. Shoot me an e-mail or leave a comment and let me know if you have some really good, or really bad, interview questions. It should be kind of fun!

Back to the Real World and RAC Architecture

I find it amazing that my last post was nine days ago already. It really seems like yesterday that I was still in San Francisco for OpenWorld. To close out that thought, the Aerosmith concert was pretty good, but not great. The could have played a little longer and done some more of the older stuff... (I was really hoping to hear "Big 10 Inch"). It is pretty amazing that they are able to put on that kind of a show at that age. I can only hope to be that active.

Thursday was actually not too bad mostly because I knew I had to be up early so I took it easier at the party than in past years. I was invited to the Inner Circle event for Oracle Fusion Applications because of my involvement on the Board of Directors with IOUG. This was my first event like this and it was very interesting. Most of what we saw and heard was not new to many in the audience because they had been taking part in the testing, but to me it was mostly new. The development and testing cycle they have been using was the most interesting part to me. It seems that Oracle is taking customer input very seriously. They have had users looking at the applications all along the way and taking the feedback and putting it into the applications. Hopefully that makes a better product for all of us.

For my company, the best part I heard was the ability to run Fusion apps along side existing applications. We are looking at adding some new modules to PeopleSoft and also bringing in some project management software. From what they showed us, the Fusion applications are built so we could bring in HR and Primavera on Fusion and seamlessly integrate with our existing PeopleSoft Financials system until we are ready to move that over. This sounds pretty nice and makes the switch seem easier than doing a full "fork lift" migration. I closed the trip with a PeopleSoft on RAC presentation and a late night trip home. All in all OpenWorld 09 was a great experience. Reconnecting with many people and putting faces to the phone voices make the trip worthwhile. Adding the content and the new people to be met are icing on the cake!

So, back to the real world. Part of the reason for attending the PeopleSoft on RAC session is that we are currently looking at replacing/upgrading our database infrastructure for PeopleSoft. I have spent part of the last couple of days researching and pricing a few different options for RAC. We currently have a shared HP Rx6600 that has eight cores and is virtualized. The database only has access to four cores and we are almost at capacity. The original plan was to purchase another Rx6600 and continue on that platform, but we are unsure of our growth over the next few years and think RAC would be a better option to give us the ability to scale as needed. That and the fact that the Rx6600 is Itanium are leading us to look at Intel based servers running Linux.

My main research points now are whether to go with more sockets with fewer cores or fewer sockets with more cores. I can see the pros and cons to both and will probably end up with fewer sockets with more cores, again because it gives us more flexibility for growth in the future. The other question is the interconnect. I'm sure gigabit ethernet would be okay, but I want to make sure there are no issues moving to a brand new platform. I'm including Infiniband switches into my design and figure I can always look at something less if we need to cut costs.

I have had to architect new solutions in the past and I always feel a little like I am putting my career on the line. That is where good research and good professional networks come into play. As long as I can back up my recommendations with multiple sources I am comfortable that things will work out. The user groups help here quite a bit (you didn't actually think I'd finish this without a user group plug). I have contacted three different people over the last couple of days for advice on my architecture. All have given me good advice and other sources of relevant information. I still haven't submitted an abstract for Collaborate 10 yet and they are due by midnight. Maybe this architecture research would be a good option...

Wednesday at OpenWorld

So, today has been a busy day for me. Several meetings with Oracle execs. IOUG is very fortunate to be supported by such great people within Oracle. The day started with a breakfast meeting with a Director in the BI area. He will be supporting us in our regional BI efforts. Mark is totally supportive of what we are doing and will be helping to promote the event within Oracle.

My next meeting was with a senior manager in Oracle North American User Groups. I have worked with Tina over the last six months on our regional user group programs. She is totally supportive of us and we had a great meeting talking about where we are going with the programs and how to better support the members in the regional areas. This support structure within Oracle has been great for IOUG as we grow our membership. They give us the support we need while letting us remain independent. Excellent partnership...

IOUG president Ian Abramson and I met with a database development director to talk about offerings in the DBA developer space. Oracle has been working on some specific training for DBA's to introduce them to some of the development tools. This training is also good for developers, but in general they are targeting the training to the database development tools. APEX, PL/SQL, JDBC, C, C++, etc are all on the radar screen. Oracle feels they are losing some database development because people are setting up smaller database environments to do the development and then going to production with those instead of moving to the enterprise RDBMS. This makes complete sense and the IOUG can work very will with this team to promote the training and hopefully even deliver some of it.

I did make it to one session today which was PeopleSoft BI Tools. This was pretty good and was timely for me since we are looking at some PeopleSoft BI options at my "real job". There are some great tools out there that can be implemented quickly without building a bunch of new stuff. I also went to my first "unconference" session ever. This was on Oracle Warehouse Builder and even though a lot of it was over my head it was great information.

Larry seems to be finally winding down... wait, maybe not. Anyway, tonight is the big party. I'm looking forward to some more networking opportunities. Last night I was at the Oracle Publisher's reception and met someone from Land's End in Dodgeville, WI. He is an Oracle Press author and wants to get involved with the Wisconsin regional Oracle users group. Amazing how small the world really is and part of why I like being here. Limitless opportunities to meet with peers.

I should be closing out tomorrow as I head home to the family. A couple days away are nice, but it is always better to go home to Mary, Grace, and Abigail. See you tomorrow Sheetz girls!

OpenWorld Monday...

Day two for me starts in the keynote with Safra Catz and Charles Phillips. We just had the intro from Judy Sims, the Chief Marketing Officer, and not it feels a little like a rock concert! Always interesting to see the new announcements and directions. Charles has been supporting the user groups a lot and has been at Collaborate for the last several years. It still amazes me the number of people they cram into this room. Great energy for a Monday morning.

Today is presentation day for me along with meetings with a couple regional user groups and special interest groups. These groups are the back bone to the user community and I encourage anyone that is interested in Oracle technology to find your local group and get involved. I'm looking forward to meeting these leaders and finding out how the IOUG can support and work together. My presentation this afternoon will be fun assuming the internet connection in the room works. I have screen shots for my demo, just in case, but that is far less interesting.

Safra Catz is talking about the amount of money they spend on R&D to help us get the right software that works. Interesting how they spend all this money to work on R&D and testing, and we still have lots to do when it comes to us. Charles just gave a shout out to user groups! They actually do listen and work with us so get involved and you can make a difference in the direction of Oracle. More later... demos coming.

OpenWorld Day 1 - User Group Sunday

What a great start to OpenWorld. The User Group Forum has been a huge success. Great content and discussion around many Oracle technologies. My day started meeting new people and talking about the IOUG at the User Group Pavilion. If you need more information about any of the user groups, stop by the pavilion and check out the offerings. Linux and virtualization are two hot topics and the expert round table session proved that. We had two people from the Oracle Linux and VM teams that answered many questions about the environments. We also had several users on the panel that have experience implementing Linux in both virtualized and non-virutalized environments. There are several more sessions this week so check them out!

My other session this afternoon was a Town Hall meeting with the Oracle Software Security Assurance team. What a group of security minded people these guys were! Critical Patch Updates, My Oracle Support, and PSU's were all hot topics. Watch for announcements from IOUG for some future follow-up webcasts. We will probably be going more in depth into My Oracle Support and how to read the risk matrices for CPUs.

I'm off to listen to Larry Ellison and Scott McNealy. Should be in interesting presentation. More to come tomorrow. I have my PeopleSoft/Grid Control plug-in presentation tomorrow. I'm looking forward to a big crowd and some more good discussion. If you are around Moscone West tomorrow, stop by the User Group Pavilion and say Hi!

Oracle Critical Patch Update Questions? #ioug #oow09

Ever since Oracle started a quarterly patch cycle for security patches, it seems that people are more confused than ever. Should I be installing these things right away? What about testing? Are there functionality patches that are "snuck" into the security patches? My company has a four month testing cycle and it would be a full time job just to keep up with quarterly patches so can I just do them once a year?

These are all questions that can be asked and answered at the IOUG/Oracle Software Security Assurance Town Hall. We will have Senior Managers and Directors from all the security teams to answer these and any other security assurance questions. This does not mean that they will be trying to sell you new tools, but rather will be there to give you as much information as they can about security and how Oracle is trying to make all the software as secure as possible.

Have you wondered where the Configuration Manager information that is gathered goes? Have you not allowed the installer to gather information or include your e-mail because you don't want a more spam? Make sure to come to the session and ask how this information is really used.

I will be moderating this session on Sunday, October 11 at 10:30 in room 2003 in Moscone West. This will be an excellent opportunity to discuss any Oracle security questions with the people that know, or can find, the information. The database, applications, middleware, and My Oracle Support teams will all be represented. Check out the session information here. Make sure to reserve your seat through Content Builder today and get your questions answered!

Clusterware Uninstall

Sometimes I really miss the old days. Of course, someone my age saying that really doesn't mean much compared to someone that lived through World War II. When I started in IT there was no www, texting, cell phones in your pocket, or USB coffee warmers. My current lament is remembering the Oracle installer when it was all command line, java was another word for coffee, and an uninstall on Unix meant "rm -rf".

Where is this going? As the title of this post suggests, I had to recently uninstall Oracle Clusterware. I am working on a project to bring a new application on board and we are using an active/passive failover model using Oracle Clusterware. This model is something I have worked on several times and did a presentation at Collaborate 08 on the topic. So, I was going along installing away, having installed OCFS2, created voting and OCR disks and then installing Clusterware. The initial install went great until I realized that I had installed into a non-standard location compared to our other clusters. After the initial "Oh Crap" moment, I just figured I would uninstall what I did and install into a new home. This is where the fun began.

Of course, I realized this was not going to be as easy as ./runInstaller and remove the Clusterware install. I did not figure that it would cost me an entire day! After running the installer and removing the Clusterware home, I received an error saying that the directory could not be removed from the other node. No problem, I'll go remove it. So, I thought I was clean. No more Clusterware directory, no more Clusterware home in the inventory because I ran the installer. Boy was I wrong.

To make this long story a little shorter, I'll get to the end of what turned out to be four install/uninstall cycles. And, by the way, looking at Metalink, the only document I could find was removing from Windows. I am on Linux and it is not supposed to be this difficult! Okay, enough whining...

1. Make sure the crs and css processes have been stopped. This should have been as easy as going to /etc/init.d and issuing the init.crsd stop and init.cssd stop, but this did not stop all processes. After all the deletions and cleanup, the only way I could really get rid of the processes was to kill them as root.
2. After issuing the stop commands, delete the init.crs, init.crsd, init.cssd, and init.evmd files from the /etc/init.d directory on both nodes.
3. Edit the /etc/inittab file and remove the lines at the bottom that start the init.evmd, init.cssd, and init.crsd
4. Remove the init.crs symbolic link from /etc/rc.d/rc0.d, rc1.d, rc2.d, rc3.d, rc4.d, rc5.d, and rc6.d. This link will start with K## where the ## is a number. Mine looks like K96init.crs and is a symbolic link back to /etc/init.d/init.crs
5. Remove the physical Oracle home directory that Clusterware was installed in.
6. Reboot! I know, it seems very Windowsish, but it was the only way to make sure all the pieces of crs and css were out of memory.

Having done all of that, I was finally able to reinstall into my new Oracle home. So, the moral of my story... make sure you are following your own standards before starting!

See you in 10 days at Oracle OpenWorld. Don't forget about User Group Forum Sunday.

Linux and Virtualization at OpenWorld #oow09

I had a very productive discussion with several people from the Oracle Linux and Virtualization team yesterday in preparation for the Linux Roundtable session on Sunday at OpenWorld. The session is going to start with an introduction to the Linux team and the Virtualization team by the directors of those technical organizations. We will then have questions and answers for them along with the remaining panel members who are all presenting topics on Linux. This should be a lively discussion and I'm looking forward to facilitating and getting some of my own questions answered.

Topics for the session will include validated configurations, why the Virtualization team chose Xen, patching cycles, how are bugs reported and fixed, and anything else the audience chooses to throw at the panel. This really should be a good combination of how Oracle thinks the technologies are being used and what they are doing to facilitate that use complimented with how real world users are doing. Don't miss the session for anything Oracle on Linux and Virtualization. The session is Sunday, October 11 at 10:30.

For the remaining Linux and Virtualization topics check out this post which lists all of Oracle's sessions. If you are coming to OpenWorld drop me a comment and let me know. I'd love to touch base with people. You will also be able to find me pretty much all day at the User Group forum on Sunday.

For now, I'm on the train on my way to Chicago for our inaugural IOUG/Chicago Oracle Users Group Regional BI Event. Should be very interesting and I'll let everyone know how it goes. Check out the upcoming events here. Upcoming sessions include Pittsburgh and Boston and plans are in the works for several others.

OpenWorld 09: T-minus one month!

Only a month remains for me to finish my presentation... yikes! Actually, the biggest problem I am having now is an open Service Request that is in development for a PeopleSoft Plug-in issue after we upgraded our PeopleTools version. Hopefully that will be resolved in the next month. It will be pretty hard to present, "look how cool this is" if it isn't working properly.

Sunday, I will be part of two sessions that I am pretty excited about. Sunday is User Group Forum day at OpenWorld and I promise there is something here for everyone. I am facilitating a Linux Roundtable discussion and a Software Security Assurance Team Town Hall meeting. Both should be fascinating. The Linux discussion will be anything to do with running Oracle technology on Linux. We will have someone from the development teams at Oracle for Linux and Virtualization, along with a some other customers that are Linux users. The Software Security Assurance Town Hall is your chance to ask all those questions you have not had a chance to ask about Security Assurance. We will have Oracle managers and directors that have responsibility for security development. Oracle Configuration Manager, My Oracle Support, Critical Patch Updates, and application security are all on the table for discussion.

Those are just my sessions (not that I'm biased of course) and all the users groups have sessions going on. Make sure to sign up for the OpenWorld Content Builder and reserve your seat for those Sunday sessions. Some of them are filling up already! There are sessions on upgrades, Exadata, BI, Spatial technologies, Content Management, RAC, APEX, Portal technologies and more. I'm starting to sound like a TV pitch man... "and that's not all!"

Of course technical content is only half the story. Who isn't looking forward to Aerosmith and being able to network and meet new people. I know I'm looking forward to catching up with some people I haven't seen since Collaborate! One way to do this for Oracle bloggers is at the Blogger Meetup that Alex Gorbachev has picked up the organizer responsibility for (thanks Alex!). Should be a good opportunity to put a face with a blog.

I better go see if I have any updates from Oracle Support... hint, hint! See you all in San Francisco.

Why can't I add a unique index?

Adding unique indexes once data is already loaded into a table can prove to be a bit of a headache. The question of how to clean up duplicate keys is one that I have had posed to me many times and it seems somewhat simple. I guess anything is simple if you already know how to do it, so to that end, I thought I would write my solution.

This became an issue again recently because the UNIX admin where I work was creating a database to store system statistics. He has daily flat files that he wants to load and knew the basics of creating a database. As he was loading the data it occurred to him that he did not want to be able to load the same file twice and he decided to try to add a unique index. Funny thing was that what he was trying to prevent had already happened and he received an error trying to create the unique index. His next stop was my desk to ask why the error was happening and how to fix it.

The first thing we needed to do was to figure out which rows were duplicated. The following SQL gave us the date that was loaded twice:

select collection_date, server, count(*)
from system_stats
group by collection_date, server
having count(*) > 1;

This gave us a list of machines for one date with 2 rows each. Now we knew what date was loaded twice, but since they were exactly the same, how do we get rid of one row for each key? This took creating a temporary table to hold the records we didn't want to delete.

create table hold_records as
select distinct * from system_stats
where collection_date = ;

Checking the hold_records table shows us that we have exactly half the number of records for that date and each server only has one record for that date. Now we clean up the data for that date from the main table.

delete system_stats where collection_date = ;

Now we put the data back:

insert into system_stats (select * from hold_records);

Voila! One record per server for that date and the unique index could be created. Again, this is not really rocket science, but I am surprised at how many times I have been asked to do something similar. Hopefully it will save someone else some time in the future.

More Manageable or Different Management?

With every new release of the Oracle Database, we are told that there are new manageability features to make our lives easier. Larry Ellison even told managers at one point that they wouldn't need a DBA anymore. I have been working on implementing the PeopleSoft Plug-in for Enterprise Manager and wrote a paper about it for Collaborate. I am presenting the paper again at OpenWorld and have been updating it with new information. This is where the question that is the title of this entry comes in. Do these tools make things more manageable or just force us to focus on a different kind of management?

Implementing management tools is only half the story. If monitoring was as easy as run setup and you're done, I wouldn't have needed to write a paper or do two versions of the presentation. What to monitor, what to alarm on, what level to alarm at, and who needs to be notified based on what the alarm is... All this functionality is built into the tools, but figuring out exactly how to set it all up is the trick. On some level I feel like writing shell scripts might be easier. Tools become a better bet when there are changes to the environment. Having one location to go to add new environments or take care of changes, does start to make tools worth while.

As with most application implementations, that is only half the story. Security has also been a challenge to set up because of the normal implementation of Enterprise Manager. Extending one tool, that is usually used for one purpose, in this case the database monitoring, to include other purposes, such as application monitoring, brings to light many short comings in the original implementation. In our case, our DBA's were Super Adminstrators with access to all targets in Enterprise Manager. Of course, from an application adminstration perspective, this causes separation of duties issues. So back to the security drawing board. Once this is designed properly, it is not that difficult to maintain, but building the proper design takes time.

Bottom line for me is that I still believe that monitoring tools and new manageability features make our lives easier in the long run. The moral of the story (and my presentation) is to plan up front and make sure you have a good design in place. I'm still working on my implementation because I didn't know what I didn't know when I started. Hopefully my paper and presentation can save someone some time. Even more interesting will be the new manageability features introduced in Oracle Database 11g Release 2. Hopefully more on those to come...

IOUG at Oracle OpenWorld

So we are about nine weeks out from Oracle OpenWorld 2009 and planning is well under way for the Users Groups. Several years ago, the Sunday before OpenWorld opens became Users Group Sunday. This year there is more participation than ever and an even higher level of technical content. The Independent Oracle Users Group has seven dedicated rooms and two shared rooms with the Oracle Development Tools User Group. Topics include 11g upgrade, Business Intelligence, Exadata, RAC, Content Management, Spatial Technologies, SAP and much more. Most of the Oracle Users Groups are represented on Sunday so make sure you include the Sunday sessions as you are making your travel reservations for OpenWorld.

Besides the Sunday program, the Users Groups also had an opportunity to choose some of the non-Oracle employee technical content for the regular conference sessions. Hinting, compression, Critical Patch Updates, 11g new features, 11g upgrades, Warehouse Builder and many more sessions were actually chosen by IOUG. Seasoned presenters, authors and Oracle Aces are all included in the line up.

And to top it all off, if you would like to get all of this for free, here is an option if you are lucky enough! Review the new Oracle Support here and be entered to win a free pass.

More details about some sessions of interest as we get closer. Hope to see you all in San Francisco in October!

Funny T-shirt or Life Lesson?

I think most men have that one t-shirt in their dresser that their wife has said, "that t-shirt is falling apart. Why don't you get rid of it?" I think I have three or four, but one of my favorites is an old Embarcadero Technologies shirt that has a picture of a woman dressed in black crying over a casket with someone tapping her on the shoulder. The caption says something to the effect of, "I know this is a bad time, but did he happen to tell you his database admin password?" I always thought it was kind of a funny shirt given how that is always the worst case scenario that we as database administrators can think of. I used to have a manager that would tell us to document our processes and procedure just in case someone got hit by the "chicken truck".

Unfortunately, I experienced this in real life last week. One of my coworkers passed away unexpectedly last weekend. I had just begun to work with him recently and so I really didn't know him that well. He was actually at an Oracle class when he got sick and had to come home. I had been working with him to help him better understand the Oracle database. In fact, I was supposed to be helping him on a new project when he came back to the office. Not surprisingly, there were some tearful eyes and heavy hearts when we all got the news on Monday morning.

Once I got over the initial shock of wondering how someone roughly my same age would go from relatively healthy to passing away, the inevitable question became, "what about supporting the organization during this sad time?" Our organization is split up so the databases that my coworker was supporting, I had never seen before. For the most part, I was able to figure out what was going on and make sure there was enough disk space, backups were running, and most things seemed in order. What we did figure out though is how much documented standards, processes, and procedures would have made the transition easier. One thing I will be discussing with management is consolidation of tasks and a set of standards that can be documented and shared with the team.

The moral of the story is pretty much what most of us already know. Document everything and make sure that the documentation and passwords are in a safe, secure place that is accessible to those that need them.

God speed Kurt... you will be missed.

Voting DOES Make a Difference... In the Oracle Community

Around election time, it is not uncommon to hear someone say, "I'm not going to vote because it never makes a difference." In these uncertain times, my response would be, "when is your vote more important?" Leaving politics aside, there are a couple places that your vote makes a difference right now!

User Group Survey
Several user groups, including the Independent Oracle Users Group have teamed with Oracle to find out how important your user group experience is to you and your company. The purpose of the survey is to show Oracle that their support of user groups is important and to allow the user groups to be able to better serve you, the member. Take a few minutes and let the community know your thoughts on user groups. You have until Friday, July 3 to make your voice heard. Take the survey here:

http://www.oraclesurveys.com/se.ashx?s=251137454671BECB


Oracle Mix Voting
Oracle OpenWorld planning is in full swing for the show in October. Did you know that you can have an influence on the content that is provided? Create an account on Oracle Mix if you don't already have one and vote for sessions that you would like to see. The link to vote is below. I voted for six sessions so far, but haven't looked at all of them. Check it out and make your voice heard so the conference is the best experience for you.

https://mix.oracle.com/oow/proposals


OpenWorld Sunday
Speaking of OpenWorld, while you are registering and making your travel arrangements, don't forget about the User Group Forum on Sunday. Several of the User Groups are presenting some of their best technical content. IOUG alone is participating in nine rooms with five sessions in each. Learn about everything from RAC, Linux, BI, and Exadata to Content Mangement, BEA/Portal technologies, Spatial technologies, and Application Express. Check out all the session on Oracle's Content Catalog at the link below. Search for Session Type of User Group Forum and SIG Meeting to find the content on Sunday. Make sure to be at OpenWorld early and take part in some great technical content.

http://www35.cplan.com/cc221_new/newCatalog.jsp

Monitoring the Monitor

Occasionally we hear about government officials that are supposed to be watching out for our good and instead look out for their own. When this happens, people will sometimes ask, "Who was watching the watchers." I was reminded of this last week when our Oracle Enterprise Manager Grid Control server lost network connectivity and we were not notified. A very large thunderstorm passed through and we also found out the next morning that we had switched to generator power at least three separate times during the night. Sometime during one of those switch overs, the network interface lost contact.

Of course we did not realize it was down for about 14 hours. The company uses HP's Network Node Manager which raises alerts to the on-call help desk person, but for some reason, no call was made to the system administrator that the server was unavailable. Not only do we use Grid Control to monitor all of our Oracle and People Soft systems, but we also use it as a centralized job scheduler. Once the server was available again, I figured out that we missed about 150 jobs. No backups, no disaster recovery log maintenance, no nightly application maintenance jobs, no nothing for fourteen hours. The better part of the next day was spent running backups and getting our Data Guard instances caught up since the archivelog destinations filled up.

So now what? We considered writing a shell script and using cron on one of the production servers to ping the Grid Control server and e-mail if it was not available. Obviously, this is a bit "patchwork" but should at least tell us if the server is unavailable. Another option would be to set up Network Node Manager to page the system administrator directly if a server goes down. This would be my choice, and I plan to work to get this implemented, but at this time, I don't have access to that system. In the meantime we decided to use our test Grid Control system (yes, we have one) to monitor the production Grid Control server. Of course this means a separate Agent installed on the production server, but at least we will have visability into the system should something happen agin.

The question through all of this is how much is enough? I guess you could ask the same question about watching our government representatives... although I think they need to be watched even closer!

Collaborate 09 - Wednesday

I wasn't able to write an update yesterday, but not because there wasn't anything to write about. On the contrary, I was so busy that I didn't have an opportunity to sit down long enough to write. The week has been very productive so far and will continue to be. Of course, everyone is looking forward to the party tonight and Universal Studios Islands of Adventure.

A few things that I didn't cover last time. Sunday evening we had the IOUG SIG Welcome Reception. This was very well attended and was a great opportunity to meet some new people and reconnect with people I haven't seen in a year. The Special Interest Groups each had a table and many people were interested in what the SIGs are doing. I was talking to many people about the Linux SIG and we should get some new blood into the group. Many thanks to SAP for sponsoring this event. We are always looking for vendors and friends that are willing to sponsor these events for us and SAP always seems to come through.

As one of the new IOUG Board of Directors members this year, I had my orientation yesterday to learn some more about my new responsibilities. I'm looking forward to the challenges and exciting times over the next two years of my term. My role on the Board will be Director of RUGs and SIGs. Joining these two groups under one Director makes a lot of sense. We should be able to jointly advertise for both groups as well as draw on each other's experiences and resources. Between the RUG Leader meeting yesterday and the SIG Council meeting this morning, I can already tell there is a lot of energy and great ideas from both communities.

The technical content this year is as good or better than it ever has been. All the new features of products and the new stuff people are doing with old things have been very interesting. Of course I have also been immersed in SIG meetings and Board of Directors meetings. I'm now in the last session of the day and learning about recovery scripts. An important topic, no doubt. It is very hard to believe that tomorrow is the last day already. I'm looking forward to a great time tonight with new and old friends. My presentation is finally tomorrow at 11:00. I'll be talking about the PeopleSoft Management Pack for Grid Control. I'm the very last session of the conference so I have a few giveaways to try to lure people in. Actually, I'm hoping that will get me some good reviews on my surveys!

See you on the roller coasters!

Collaborate 09 - Day 1: Charles Phillips

What a great start to Collaborate already! Charles Phillips, President of Oracle, had our keynote presentation this morning. Some highlights from the talk include some new releases, extension of support and a demo of Beehive. Charles mentioned that PeopleSoft 9.1 should be available around OpenWorld time. Not much more on that because the big deal was the announcement of EBuiness Suite 12.1. Along with that comes 9 new products that have been updated to 12.1 and 14 production that can run on 11i10 with no upgrade required. One of the main threads that came through was Oracle's sensitivity to the economic times and that not everyone can afford to upgrade. Along those lines he also announced extended support for several products without any additional cost. These included 10gR2 database, which was extended to July 2011, PeopleSoft 8.9, which was extended to Jun 2011, and EBusiness 11.9. I neglected to get that date, but they basically added a year. There were a couple other products that were extended, so I guess it's time to check out the support grid again.

Much of the discussion covered integration and collaboration. The Application Integration Model (AIA) has been expanded to include more applications and uses Oracle Middleware and web services to integrate applications. Oracle is selling the Process Integration Packs (PIP) in order to automatically integrate the applications. These are used to let Oracle worry about your integrations. Sounds great... I'd like to see it in action.

Speaking of seeing in action, there was a demo of Beehive. For many of us, we have heard the term, but are not sure what Beehive is or does. Collaboration and projects are becoming a big focus and this helps to address those issues. The big announcement today for Beehive is the release of the Team Collaboration module that goes GA tomorrow. This allows project teams to create Wikis, team calendars, versioning and check out of files all through one server. Beehive alos includes chat and e-mail and can integrate with other e-mail solutions. The demo was pretty cool and Charles said that Oracle is running this with ten to fifteen thousand users being hosted on one server.

One interesting side comment was about Exadata. Charles was talking about Exadata and how it currently uses Linux. Then he said that the Sun acquisition will not have any effect on Exadata, but that there will probably be a Solaris version at some point... interesting, very interesting.

I am now in the Darwin Awards for IT security. Some of these examples are a little funny and even more scary. I just hope never to be a recipient! More about Collaborate later.

My (Almost Solid) Collaborate 09 Schedule

With less than a week before Collaborate 09 in Orlando, I have been working on the schedule of sessions that I don't want to miss and trying to fit in as many as I can around meetings. Every year, I have the same issue. Do I attend sessions for things I'm working on now to see if I can better my current processes, or do I look for new features and future products? This is actually a good problem to have and is an indication of the level of content that will be available.

Meetings for Special Interest Groups and now the IOUG Board of Directors take about half of my time, but that leaves plenty of time for some great session content. I will be dipping into some of Quest' International's content this year also since I am back to supporting the PeopleSoft world. Besides PeopleSoft I am trying to balance my time between new topics and improvements to current projects I am working on.

I'm especially looking forward to the Darwin Awards for IT Security and New and Improved Features in 11g RMAN. There should be several things in both of these that I can use right away back at the office. Backup scripts without restore scripts??? Who would do that? I guess that will be one session I will have to check out and make sure my restore scripts are up to snuff. Of course, my own session, which happens to be the last technical session on Thursday will be of interest to anyone that supports PeopleSoft. We are implementing the PeopleSoft Management Pack for Grid Control and my presentation covers some of our headaches and successes.

Collaborate wouldn't be complete without catching up with all the people I have become friends with in the past and making some new ones. Talk about value for the money. My contacts from past conferences have saved me a ton of time and my company some cash in the process. If you think you might be missing out because you won't be in Orlando next week... you're right! You still have two days to get $200 off the on-site registration rate and discounted hotel rates. Make sure you don't find yourself stuck in a situation that you could have learned about by attending Collaborate 09!

And when Collaborate is over.... I'M GOING TO DISNEY WORLD!

I plan on Tweeting from the conference and will hopefully find some time to write blog entry or two. I hope you follow along! Find me on Twitter... toddsheetz.

"Virtual" Test

Budgets, budgets, budgets. We all have to deal with budgets. Making sure we can pay the mortgage, make the car payment, and feed the family fit into the home budget. For anyone that is a member of an organization, such as IOUG, we may want to start a really cool project, but being a volunteer organization, the budget will not support it this year. Even sending my daughters to school has become interesting as the private school system that they attend has run into budgetary issues. Most everyone in the technology field, definitely has run into budget constraints at work. In my case, the most recent budget "issue" has been creating a test system for our latest project.

A project was started to consolidate some remote MS Access databases into a centralized Oracle database. This is a vendor application and either installation method is supported. As the application becomes more mission critical, however, the decision was made (and the correct one, in my opinion) to consolidate to our Network Operations Center. The issue that came up though was dollars. When the original budget was written for the project, dollars for a test system were forgotten. The production system was budgeted as a two-node cluster to support two different application databases. This was fine and should support what we want to accomplish, but test was completely overlooked.

So, how do you create a test cluster to support a production implementation when dollars are limited? In our case, the decision was made to purchase one physical server and user Oracle VM to create a test cluster. Not exactly the same as production, but at least we will have somewhere to test out operating system, database, clusterware, and ASM upgrades and patches before going directly into production.

Of course, we have had issues. There is a larger VM learning curve than we expected especially as it relates to storage. We are also feeling our way through resource allocation. Originally we underallocated CPU resources and couldn't install anything. The virtual guest machine showed that the CPU was pegged, but the physical host showed no activity. Allocating an extra virtual CPU took care of that issue, but we still had issues installing and configuring ASM. The normal installation of 11g and configuration of ASM was hanging and showing full CPU utilization, although it was creating the instances. My coworker has spent a couple days with Oracle support trying to figure things out and finally ended up creating everything manually instead. Just one instance of how virtualized environments are not quite as seamless as the are touted to be.

In any event, we have the viritual servers up and running. So now we need to create the databases and away we go! That is just one example of how, if necessary, budgets can be saved. Of course, we haven't proven that we can support any kind of test load on this architecture, but we should at least be able to see if the applications still function after an upgrade or patch.

One final plug, by the way, for the IOUG Forum at Collaborate 09. If you are looking for more ways to save budget dollars, there are lots of tips and tricks you can learn from your peers. I'm hoping to learn more about virtualization and how we can better tune our test system. Since budgets are tight everywhere, make sure you register online before April 30 to save $200. Also, the conference hotel rates have been discounted. There is not better bargain for training than learning from people that have actually used the technology...

Collaborate 09 - Early Bird Registration and Special Offer

Okay, so I know I already posted today, but I don't think I put enough information about Early Bird Registration for Collaborate 09. You only have one more day to save $400 and get some extra stuff by using one of the special registration codes.

Register for the IOUG Forum by this Thursday, April 2 at http://w3.ioug.org/COLLAB09/LinkedIn using promotional code AD01 (choose this code during the demographic section – step 4 – of the registration process). You will receive a free Oracle Technology Best Practices Booklet and have your name entered into a raffle for an Apple iPod Touch. Of course, you also get access to the entire conference – more than 1,600 sessions! Oracle database, RAC, PeopleSoft, eBusiness Suite, Grid Control, and much more all in one place. All this training for one low price will save your company money by not having you go to multiple classes and you will be able to put the lessons learned into practice as soon as you get back to the office - and don't forget the networking.

Okay, enough marketing! I only blog about this because it is very important to me and I have benefitted from it for many years. I promise, you will not be disappointed!

Hope to see you in Orlando and make sure you get that registration in by tomorrow!

April Fools Thoughts...

I am in a RAC Workshop at the Oracle office in Milwaukee today. We are still in the set up phase so I have a few minutes. This should be either a good follow-up to our recent Clusterware install, or a rude awakening that we need to start over! Let's hope for the former.

Yesterday and today we were in the process of installing a kernel upgrade to our cluster nodes. I have used Clusterware and OCFS on Linux in the past, but never with ASM. The trick was figuring out the order between installing the kernel upgrade and the ASM and OCFS drivers to match the kernel. To add to the trouble the QLogic driver needed to be reinstalled. Lots of learning experiences! The pleasant surprise to me was that once we got everything upgraded on the first node all the pieces, including Clusterware, started normally, even though node two was not yet upgraded. This is a nice upgrade from my last experiences with all this when we had to take node two down before OCFS and Clusterware would start on the upgraded node. Nice to know that future upgrades should allow for much shorter downtime during the failover.

A reminder that early-bird registration for Collaborate 09 closes tomorrow. Hopefully everyone can make it down to Orlando in May. As a member of the Conference Committee, we are hard at work reviewing papers and presentations. The technical content should be excellent as usual. I have been reviewing papers for the Achitecture track and have read some great papers. Since I finally finished my own, I have more time to read others. Now I just need to complete my presentation... details, details.

IOUG members are reminded that the Board of Directors elections are open until Saturday. This is a bit of a selfish plug since I am running for the Board. Most importantly, make sure you vote. A vote for me is very appreciated! Next year's Board promises to be a very good group of people and will continue move IOUG forward.

The workshop is starting... better go pay attention!

Clusterware and PERL

A couple weeks ago, I wrote about my new project at work. We are implementing single instance databases protected by Oracle Clusterware. Eventually, the thought is that we will upgrade to Real Application Clusters, but for now we should be able to support the applications with single instance databases.

This week we completed the set up of the cluster after a few technical delays. Our final hurdle seemed to be starting the database using Clusterware. Part of the registration process for a resource in Clusterware is to tell Clusterware what script has the actions available for that resource. Oracle provides action scripts for resources such as listeners, databases, and Enterprise Manager Agents in a couple white papers that need to be copied out and implemented. The action scripts that are provided in the papers are in the form of PERL scripts so they are portable. PERL is one of those things that I have always meant to learn, but never really made the time to pick it up. The database action script was failing when we tried to start the database, but we were able to start the database on both nodes without using Clusterware.

With some help we were able to add some logging to the script and sent some of the variable values to a text file. Using this method we were able to narrow down the issue to the check subroutine in the script. Further analysis narrowed the problem down to the following line of code:

$check_proc = qx(ps -aef | grep ora_pmon_$ORACLE_SID | grep -v grep | awk '{print $8}');

When I cut the line out of the script and ran it from the shell, it returned exactly what was expected. When we logged the output from in the PERL script, it returned the entire line from the process. For some reason, the awk part of the command was not being executed. Finally, I realized that PERL was treating the $8 part of the awk command as a variable, but since there was no variable for 8, the awk was not returning the eighth field as expected. A simple backslash in front of the $ took care of the issue:

$check_proc = qx(ps -aef | grep ora_pmon_$ORACLE_SID | grep -v grep | awk '{print \$8}');

About the same time that I figured this out, the person I had been working with at Oracle informed me that I had been using an older version of the script. They have since replaced the line with a better option:

$check_proc = qx(ps -ae -o cmd | grep -x ora_pmon_${ORACLE_SID});

Obviously this is much cleaner and accomplishes the same thing. So I learned two things from this experience. Number one, always check the dollar signs in a PERL script and number two, make sure you have the latest version of the script!

Why Collaborate?

I have mentioned in previous posts that I will be presenting a paper at Collaborate 09 in May. This will be my third year presenting and I have found it very rewarding. I attended IOUG events back when the conference was called IOUG Live, and it has only gotten better since I have been involved with presenting. So why should someone go to Collaborate instead of taking an Oracle class? Why should you fight to go when training budgets are being cut? I have several reasons.


Content
Collaborate is one of the few places that you can find the best technologists in the Oracle community all in one place. These are people who are really doing the work and managing the projects to implement and support Oracle technology. Go through your list of Oracle authors and chances are they will be there. Tom Kyte, Rich Niemiec, Michael Corey, Ian Abramson, Michael Abbey, Carey Millsap, Jonathan Lewis... and the list goes on. There will be 350 sessions and that is just from the IOUG Forum. Add in OAUG and Quest International and the options are almost endless.

Are you having issues with Real Application Clusters? Can't quite figure out how to get PeopleSoft monitored with Enterprise Manager? Need help designing a process to patch eBusiness Suite? Starting your first BI project and don't know what BI means or what it has to do with stars? Answers to these questions and more can found at Collaborate. Many times I have been sitting in a session when the speaker said something and I thought, "so that's how you do that!"


Networking
I already listed some of the well known authors/presenters that will be in Orlando this year, but that's really only half the story. Ask the Experts sessions, topic specific orientations, the IOUG/SIG Welcome reception, "Tuning" Open Mic Night, lunches and receptions are all great places to meet new people that are doing some of the same things you are doing. Personally, there have been many times that I have been stuck on an issue, or just wanted to bounce an idea off of someone, that I have e-mailed or called someone I have met at Collaborate or through the IOUG. The network of people that I have connected with has been a huge time saver for me and a money saver for the companies I have worked for.


Cost
Recently I did a comparison between what it would cost for me to go to and Oracle instructor led training course instead of going to Collaborate. To just attend the conference, you could get away with traveling Sunday to Thursday and only pay for four hotel nights at one of the Conference hotels.

Collaborate Early Bird Registration:	$1,335
Airfare:	$200
Airport transfer:	$29
Hotel - Four nights@$140 + tax:	$625
Food:	$100
Total:	$2,289

Oracle Instructor Led Class:	$3,000
Train Fare:	$44
Cab Fare Roundtrip:	$40
Hotel - Four nights@$160 + tax:	$739
Food:	$125
Total:	$3,948

As you can see, you can save at least $1700 by going to Collaborate instead of an Oracle class. And that is if you have access to taking the train to an Oracle education site on Monday morning. If you have to fly, there would be added cost for the flight, plus an extra hotel night. Collaborate is four days and the Oracle class is five days, so that is part of the difference.

If you add a Sunday University Seminar for $459 and one extra hotel room night at $155, you would still only be spending $2903, a full $1000 less. You could also add a Thursday University Seminar for just the cost of the session and not add to the travel budget. The University Seminars are targeted deeper dive training. I would highly recommend trying to go to at least one of them.


Why Collaborate?
You can now see what I Collaborate! You could say it has become a little bit of a passion, but it is something I very much believe in. Check out the registration link at www.ioug.org/collaborate09 and look for the top ten reasons to attend. Also, you can find a proposal to give to your management on why it is important to be at Collaborate. Hope to see you in Orlando!

Until next time... stay clean!

New Clusterware Project

For the last week, I have been working on our newest project. We are consolidating an application that is currently in pilot mode across several locations. Some of the current databases are SQL Server and some are Oracle. We were brought into the project a little late, but have recovered nicely to work out an architecture that we think will be a good starting point for growing this environment.

I have learned a lot about project teams and technology through this process. I work for Veolia Environmental Services and if you have heard of our company, you know it is huge. We are starting to do Information Systems support for some of our sister companies, and that alone is proving to be a challenge. Trying to meld different corporate cultures is bad enough and then add in the technology layer and it can be really fun! I may cover the culture clashes some other time, but for nowI thought it might be interesting to go over the technology.

We started planning on using Oracle Standard Edition database and Real Application Clusters running on Oracle Enterprise Linux using HP hardware with Intel x86-64 processors. Many of our discussions centered around the version of Oracle and whether or not to implement RAC. We made the decision to stick with Standard Edition even though all of our current implementations are Enterprise Edition because of budgetary concerns. The plan is to move to Enterprise Edition in the future so we can standardize our licensing, but for now, we have a limited budget and are implementing Standard Edition. We also decided to forgo RAC and user Clusterware to protect the databases using active/passive clustering instead of active/active. Our timeline was very short and I have used Clusterware in this fashion before so this helped to shorten the timeline, but we also don't really know how much traffic we are talking about so RAC may not even be necessary.

So we have Clusterware installed and running, Automatic Storage Management is running and storage is allocated, the RDBMS is installed and patched, and we are ready to create the databases. Next comes the fun part... pushing power buttons and pulling power cords to make sure it all works as planned! This has been an incredible learning experience and should continue to be. Next is building our test environment using Oracle VM! Let me know any comments if you have had similar projects or want to chat about what we're doing.

Until next time... keep learning! Oh, by the way, I'm sure there will be lots of good sessions on Clusterware, RAC, Linux, etc at Collaborate 09 this year.

Oracle Magazine Interview

Today was kind of an exciting day for me. I was interviewed via webcam and Skype for Oracle Magazine. They do a Linux issue every year and they have started doing a feature on the user community. I am the current president of the Linux Special Interest Group (SIG) for the Independent Oracle User Group and they asked if they could interview me about the SIG and our relationship with Oracle. I was contacted by Jeff Erickson at Oracle Magazine through Rich Schwerin, who I mentioned in an earlier blog post. Rich is the Senior Product Marketing Manager in the Linux and VM group at Oracle. We talked for about a half hour about what the SIG has done in the past, our future plans, and how Oracle is doing in the Linux and support areas. I won’t ruin it for you and you’ll have to check back to find the link to the video, but I really appreciate the opportunity to talk about the IOUG and the Linux SIG.

For part of the interview we discussed personal and professional opportunities that come from involvement, not just membership, but involvement, in the user group community. Personally, I have made some great friendships and have people that I talk to, either on the phone or electronically, regularly. Professionally I have made some networking relationships that have saved me several times. This is where involvement really comes in. I started with IOUG as a member attending the conferences. That was great, and I learned a lot, but it was always lacking a little something for me. Once I started actually getting involved and taking a leadership role in the SIGs, the experience changed. The interview also covered my experience with Linux and the relationship with Oracle. At my previous company, we implemented Oracle Clusterware on Red Hat Linux. Those networking relationships really came in handy there! Now, my coworkers and I are embarking on a Clusterware project using Oracle Enterprise Linux. I'm looking forward to getting back into it as well as having something to share with the Linux SIG.

So, I will post the link to the video in a couple months. Hopefully I didn’t look too geeky. In the meantime, make sure you are getting the most out of your career by getting involved. And don't forget that Collaborate 09 is coming up in May. There will be several sessions on Linux and over 350 sessions total just from the IOUG Forum. IOUG is always looking for good volunteers that are willing to step up and take a leadership role. The SIGs are a very good way to do that. Who knows… maybe next time you could be the one being interviewed for the magazine! Speaking of that, make sure you check out IOUG President Ian Abramson’s blog today. He is also talking about being interviewed by the media. He also has a good post about some of the inner workings of the IOUG Board from yesterday.

Collaborate 09 Paper: PeopleSoft Management Plug-in for Oracle Enterprise Manager (aka Best Laid Plans)

“The best laid plans of mice and men often go awry.”

I’m working on writing my paper for Collaborate 09, and for some reason, this phrase keeps coming back to me. I actually looked up this quote and it is often misquoted, including in my opening sentence. The quote is actually from a poem by Robert Burns, called “To a Mouse” and the quote is actually, “The best laid schemes o’ mice an’ men, Gang aft a-gley.” Burns, incidentally, also wrote Auld Lang Syne. My best laid plans included actually having my white paper and presentation completed for Collaborate 09 before they were due. This year my presentation is “Implementing the PeopleSoft Enterprise Environment Management Plug-in for Oracle Enterprise Manager.” I did get an early start, but have been working on this for about six months and have been unable to complete the project.

Issues started almost immediately with the installation process. We are an HP-UX shop running on Itanium CPUs. The installation of the plug-in to the Oracle Enterprise Manager Management Service failed with a password issue. After several weeks, Oracle determined that the installer was having issues on Itanium. They created a patch for the installer that allowed me to install the plug-in and I was finally on my way. You will have to wait for my paper and presentation for the details of the installation and setup, but I did run into a few more road blocks that fed into the “Best Laid Plans” theme.

One thing that got me was that I had installed the Oracle Management Agent on my PeopleSoft servers before starting to do the research for the PeopleSoft plug-in. Big mistake! I was hoping to monitor the up/down status of the hardware and the available space for the file systems. This worked well, but then I found out that the installation of the agent for the plug-in is required to be owned by, or at least in the same group as, the user that owns the PeopleSoft installation. So after several uninstalls and reinstalls, I was ready to go again. Of course then there was the issue of what user to use when discovering the PeopleSoft targets. The documentation is unclear and I tried several options. Bottom-line is that while the Management Agent can run as a centralized user that shares the group with the PeopleSoft installation, the discovery has to be done by the actual software owner. This was an issue only because we have several environments hosted on the same server in our non-production environments. That was a long paragraph and hopefully makes some sense. If not, I wrote it much better in the paper so you can check it out at Collaborate.

My next challenge had to do with how PeopleSoft identifies its databases. This is something I had no clue about and caught me off-guard. I went through and, with the help of the PeopleSoft Administrators, added all the non-production PeopleSoft targets. One of the cool features of the plug-in is that it allows you to create a system for each environment and a service to measure the login time and make sure the web site is available. When I started to create systems for each environment, Enterprise Manager tried to connect all the components to one database. After further research I found that PeopleSoft stores a GUID in the PSOPTIONS table that needs to be unique for every environment. Since we refresh our test environments with production, all the GUID fields were the same, thus the reason that Enterprise Manager tied everything together. The fix for that is to shut down the application server, set the GUID to a single space in PSOPTIONS, and then restart the application server. The application server then assigns a new GUID and all is wonderful… Unfortunately, the developers, testers, and PeopleSoft administrators are not that happy to give up the environments for outages. Plus, I had to delete all the targets that had been previously discovered and go through the discovery process all over.

At this point I am almost ready to add the production targets to the Enterprise Manager repository and really start using it like it is meant to be used. And of course, I found one more issue. We refreshed one of our non-production environments the other day. Part of the service monitoring includes a user in the PeopleSoft system that only has rights to log in and nothing else. Of course, we decided to use a different password that we could standardize for all non-prod environments, but it is different than what we used the first time. I cannot for the life of me, figure out where to change that password to make the service check work again. I ended up creating a Service Request with Oracle and apparently, they don’t know either. It has been a week and they haven’t given me a response.

Well, I better actually get to finishing the paper. I have not been able to find a lot of information about people using the plug-in so if you are, please drop me a line and let me know. I would love to trade war stories! Also, if you are not already planning on attending Collaborate 09 in Orlando in May, make sure you check it out. It is hard to beat the amount of technical content and the networking opportunities that exist at this one conference.

Until next time, and as always… stay clean!

Upcoming Webinar – Auditing Database Activity for Security and Compliance with Oracle Audit Vault

In my initial entry, I mentioned that I am involved with the Independent Oracle User Group and the Special Interest Groups. The Enterprise Best Practices SIG is sponsoring a very exciting webinar next week that any Oracle technologist should be participating in. Tanya Baccam is an instructor with the SANS Institute and Roxana Bardescu is a Senior Principal Product Director in the Database Security Marketing Group at Oracle. They are co-presenting a webinar on Audit Vault, Oracle’s new tool for capturing audit information. The webinar is scheduled for Wednesday, March 11 at 2:00 pm Central Time. Here is the abstract:

Using the native auditing capabilities provided by the Oracle Database and Oracle Audit Vault, organizations can easily monitor their databases for suspicious activities and simplify compliance reporting without costly investments in third-party solutions. In this session, you will learn best practices for effectively auditing your Oracle databases as well as how Oracle Audit Vault can proactively collect and analyze that data in real-time to generate alerts and reports that satisfy internal security policies and regulatory compliance.

I have heard both Roxana and Tanya present before and they are very knowledgeable. This should be a very informative webinar. You can register by following this link.

If you want more on security, you can really get an in depth view of database security at Collaborate 09. Tanya will be presenting a University Seminar on Sunday that is an abbreviate version of the SANS Institute’s week long Securing Oracle Database training. This will give you enough information to scare the heck out of your manager!

Check out the webinar… nothing like free training in this day! Happy auditing!

Oracle Releases Grid Control 10g Release 5 – Follow-up

I just received a follow-up to the webinar yesterday and it is now posted online. You can go to Oracle Technology Network and listen to the recorded webinar here.

After my post yesterday, I received a note on LinkedIn from Rich Schwerin, Senior Product Marketing Manager with the Linux and VM group at Oracle. I have worked with Rich in the past on a roundtable at OpenWorld 2008 and he put me in touch with a couple people for webinars for the IOUG Linux SIG. He reminded me about the new VM features that Oracle put into 10gR5. This is very cool because you can manage the entire stack, right down to the virtual hosts, through the centralized console. Add that to the Linux management pack that is included with your support license for Oracle Enterprise Linux and you really do have a top down management opportunity.

I encourage you to check out the webinar and technical information if you are a VM and/or Enterprise Linux user. The new features should help to make our jobs much easier.

Thanks for the note Rich!

Oracle Releases Grid Control 10g Release 5!

I know I said my next entry would be about my paper for Collaborate 09, but I sat through Oracle’s webinar today about Grid Control 10gR5 and decided to write about that instead. I’ve been waiting for this release because a lot of the 11g database features have been built into the Enterprise Manager interface. Real Application Testing and Transparent Data Encryption are features that are in the 11g database, but you had to use the local Enterprise Manager to use them. Now they are built into Grid Control. The release number is 10g Release 5, but this was actually supposed to have been the 11g release. Rumor has it, and presenters at conferences have backed this up, that they were ready to call this 11g, but when the BEA purchase went through they decided to hold off on the 11g release until they evaluate revamping Grid Control to use WebLogic instead of Oracle Application Server. We shall see…

The webinar from today is supposed to be posted by tomorrow, but the slides are already there. This really does look like a very nice, feature packed, upgrade. We are on HP-UX/Intanium at my company so I have to wait a bit longer, but if you are using Windows 32 or Linux/Intel, you are good to go. Check out the Enterprise Manager page on OTN and read about the new features. The Real User Experience Insight looks to be a nice feature for measuring performance at the user level. There are accelerators already created for E-Business Suite, Seibel, and PeopleSoft. This should be fun to play with since I support PeopleSoft.

Oracle is building in their acquisitions as fast as they can. One new acquisition that was mentioned on the webinar is mValent. The acquisition was just completed a couple weeks ago and Oracle is already working to build those features into Grid Control. I’m not familiar with the mValent software at all, but apparently it is supposed to make managing complex applications even easier. It should be interesting to see how all these new features for managing complex application environments fit in with already existing plug-ins. As you will see in my later post, I have had several headaches getting the PeopleSoft plug-in implemented and if these tools turn out to be easier, it may be the end of the current plug-in model.

I better get going on my Collaborate paper. Hopefully once I finish it, I can actually write the blog posting about it. Feel free to drop me a line if you like the blog or have some suggestions. Also, I’d love to hear from you if you have had a chance to play with 10gR5!

Ian Abramson, president of the Independent Oracle User Group, had a very thoughtful blog posting today regarding spending in the current economic times. Companies have to make tough choices when it comes to spending and sometimes it is difficult to make a difference as a "rank and file" employee. One thing to do is make sure you are getting the most out of what they will give you. This includes making the right choices and pushing the right areas for your career.

I have been attending IOUG events since 2000. This was after attending several classes from Oracle, a couple programming classes, a Sun Solaris class, and some project management classes. The interesting thing is that there are sessions on all of these subjects in the one week of Collaborate and after the session you can have an adult beverage (or a soda) with the presenter and ask him/her real world questions. If you are fortunate enough to have a company that still is able to send you to training, think about the best way to spend those dollars. One week on one subject... or one week on about 400 subjects. Check out the session offerings at the Collaborate 09 site and you can even download a pre-written letter to your manager helping you to sell the experience.

My next post is planned for my adventure while writing my paper for Collaborate so stay tuned. Until then... stay clean!

IOUG Relases Security Survey

Today, the Independent Oracle User Group released the results of a survey conducted of members about their processes for applying Critical Patch Updates. These patches are Oracle’s method to fix any security holes in the software. Others have described in depth the methodology and the findings, so I will include those links at the end. As for what this means to me, it very much fits into how I have been looking at CPU’s since Oracle started releasing them.

Before I talk about my opinions, I need to have a little disclosure. I am a member of the Independent Oracle User Group and was involved in promoting the survey. The comments included in this blog are my opinions and are not a reflection of other members of IOUG. Hopefully that will protect me if I tick someone off!

The CPU process is relatively simple to apply. Shutdown the databases that are running using the affected Oracle software installation, run the patch process, restart the database, and run the .sql script that is delivered. This patch process has been around for a long time for “one-off” patches and the Security team in Oracle has made it relatively fool-proof.

So why, as the survey finds, do people not install these as regularly as Oracle would like? I believe there are a couple reasons and these are somewhat reflected by the answers to the survey. First, everyone is busy keeping their environments running and implementing new projects. Most people are attempting to accomplish more within their organizations with less resources and it is difficult to find the time to perform the patching. Second, Oracle tells us that these patches have been through numerous tests, and I believe they have. Up to this point, however, people have been instructed to test, test, and test again. It is difficult to get through a full test cycle on the patches within the three months between the patch cycles. Finally, when technologists (a fancy word for those of us that are in charge of the patching) approach management about needing down time to apply the patches, management wants to know why and what are they gaining. For the most part, plugging security holes is not high on a manager’s priority list when there is a backlog of projects that need to be implemented.

How do we as technologists work through the listed challenges? I think the most important thing is to make sure you are educated. This can be done by reading Oracle’s readme files and any documentation about Critical Patch Updates. Also, conferences, such as Collaborate 09 have many sessions presented by users and by Oracle regarding the CPU process and security in general. Another point to work on is convincing management that these patches are important and have relatively low risk. We still need to run some testing, but most people will not be required to complete a full blown regression test.

Check out the survey and read what others are saying. The survey can be found here:

http://enterprisesig.oracle.ioug.org

Two other blogs can be found at the following. I will also update as any other news or commentary is posted.

http://blogs.oracle.com/security http://michelledbaunleashed.blogspot.com

The 5 Ws of My Blog

For my initial blog entry, I decided to ask myself and then answer a few questions. Who am I? Why would I write a blog? What will I write about? Where can you find my interests on the internet? When will I post entries to my blog?

Who Am I?

Ah yes, one of the most asked questions in the universe. My name is Todd Sheetz. I am an Oracle DBA in the Milwaukee, WI area. I currently work for Veolia Environmental services and support a PeopleSoft/Oracle 10g environment. I live, with my wife and two daughters, in Waukesha, a suburb of Milwaukee. I am an active member of the Independent Oracle User Group, but more on that later.


Why would I write a blog?

For lack of a better reason, why not? I figure this will help me to organize thoughts, hopefully start some discussions, and be one more area to promote groups and events that I think are worthwhile.


What will I write about?

For the most part, I plan on writing about projects I am working on and organizations I am involved with. Other “life” topics may find their way here, but those will be few. My interests, work-wise, include Oracle database administration, Enterprise Manager Grid Control, Linux, PeopleSoft administration, and disaster recovery. Besides work, I am a member of the Independent Oracle User Group and have been very involved in the Special Interest Groups. I will be posting updates about the going’s on of those groups also.


Where can you find my interests on the internet?

Besides trying to see how this blogging thing will go, I am on LinkedIn and Facebook. I will have links as I go to www.ioug.org and various SIG events.


When will I post entries to my blog?

Usually, I plan on posting when something interesting piques my interest. There are several webinars coming up that I will be participating in so I will post links and possibly summaries for those. Otherwise, it will just be when I have something that I think others may be interested in.

Until next time… stay clean!