Saturday, March 28, 2009

Clusterware and PERL

A couple weeks ago, I wrote about my new project at work. We are implementing single instance databases protected by Oracle Clusterware. Eventually, the thought is that we will upgrade to Real Application Clusters, but for now we should be able to support the applications with single instance databases.

This week we completed the set up of the cluster after a few technical delays. Our final hurdle seemed to be starting the database using Clusterware. Part of the registration process for a resource in Clusterware is to tell Clusterware what script has the actions available for that resource. Oracle provides action scripts for resources such as listeners, databases, and Enterprise Manager Agents in a couple white papers that need to be copied out and implemented. The action scripts that are provided in the papers are in the form of PERL scripts so they are portable. PERL is one of those things that I have always meant to learn, but never really made the time to pick it up. The database action script was failing when we tried to start the database, but we were able to start the database on both nodes without using Clusterware.

With some help we were able to add some logging to the script and sent some of the variable values to a text file. Using this method we were able to narrow down the issue to the check subroutine in the script. Further analysis narrowed the problem down to the following line of code:

$check_proc = qx(ps -aef | grep ora_pmon_$ORACLE_SID | grep -v grep | awk '{print $8}');

When I cut the line out of the script and ran it from the shell, it returned exactly what was expected. When we logged the output from in the PERL script, it returned the entire line from the process. For some reason, the awk part of the command was not being executed. Finally, I realized that PERL was treating the $8 part of the awk command as a variable, but since there was no variable for 8, the awk was not returning the eighth field as expected. A simple backslash in front of the $ took care of the issue:

$check_proc = qx(ps -aef | grep ora_pmon_$ORACLE_SID | grep -v grep | awk '{print \$8}');

About the same time that I figured this out, the person I had been working with at Oracle informed me that I had been using an older version of the script. They have since replaced the line with a better option:

$check_proc = qx(ps -ae -o cmd | grep -x ora_pmon_${ORACLE_SID});

Obviously this is much cleaner and accomplishes the same thing. So I learned two things from this experience. Number one, always check the dollar signs in a PERL script and number two, make sure you have the latest version of the script!

Tuesday, March 24, 2009

Why Collaborate?

I have mentioned in previous posts that I will be presenting a paper at Collaborate 09 in May. This will be my third year presenting and I have found it very rewarding. I attended IOUG events back when the conference was called IOUG Live, and it has only gotten better since I have been involved with presenting. So why should someone go to Collaborate instead of taking an Oracle class? Why should you fight to go when training budgets are being cut? I have several reasons.

Collaborate is one of the few places that you can find the best technologists in the Oracle community all in one place. These are people who are really doing the work and managing the projects to implement and support Oracle technology. Go through your list of Oracle authors and chances are they will be there. Tom Kyte, Rich Niemiec, Michael Corey, Ian Abramson, Michael Abbey, Carey Millsap, Jonathan Lewis... and the list goes on. There will be 350 sessions and that is just from the IOUG Forum. Add in OAUG and Quest International and the options are almost endless.

Are you having issues with Real Application Clusters? Can't quite figure out how to get PeopleSoft monitored with Enterprise Manager? Need help designing a process to patch eBusiness Suite? Starting your first BI project and don't know what BI means or what it has to do with stars? Answers to these questions and more can found at Collaborate. Many times I have been sitting in a session when the speaker said something and I thought, "so that's how you do that!"

I already listed some of the well known authors/presenters that will be in Orlando this year, but that's really only half the story. Ask the Experts sessions, topic specific orientations, the IOUG/SIG Welcome reception, "Tuning" Open Mic Night, lunches and receptions are all great places to meet new people that are doing some of the same things you are doing. Personally, there have been many times that I have been stuck on an issue, or just wanted to bounce an idea off of someone, that I have e-mailed or called someone I have met at Collaborate or through the IOUG. The network of people that I have connected with has been a huge time saver for me and a money saver for the companies I have worked for.

Recently I did a comparison between what it would cost for me to go to and Oracle instructor led training course instead of going to Collaborate. To just attend the conference, you could get away with traveling Sunday to Thursday and only pay for four hotel nights at one of the Conference hotels.

Collaborate Early Bird Registration:$1,335
Airport transfer:$29
Hotel - Four nights@$140 + tax:$625

Oracle Instructor Led Class:$3,000
Train Fare:$44
Cab Fare Roundtrip:$40
Hotel - Four nights@$160 + tax:$739

As you can see, you can save at least $1700 by going to Collaborate instead of an Oracle class. And that is if you have access to taking the train to an Oracle education site on Monday morning. If you have to fly, there would be added cost for the flight, plus an extra hotel night. Collaborate is four days and the Oracle class is five days, so that is part of the difference.

If you add a Sunday University Seminar for $459 and one extra hotel room night at $155, you would still only be spending $2903, a full $1000 less. You could also add a Thursday University Seminar for just the cost of the session and not add to the travel budget. The University Seminars are targeted deeper dive training. I would highly recommend trying to go to at least one of them.

Why Collaborate?

You can now see what I Collaborate! You could say it has become a little bit of a passion, but it is something I very much believe in. Check out the registration link at and look for the top ten reasons to attend. Also, you can find a proposal to give to your management on why it is important to be at Collaborate. Hope to see you in Orlando!

Until next time... stay clean!

Friday, March 13, 2009

New Clusterware Project

For the last week, I have been working on our newest project. We are consolidating an application that is currently in pilot mode across several locations. Some of the current databases are SQL Server and some are Oracle. We were brought into the project a little late, but have recovered nicely to work out an architecture that we think will be a good starting point for growing this environment.

I have learned a lot about project teams and technology through this process. I work for Veolia Environmental Services and if you have heard of our company, you know it is huge. We are starting to do Information Systems support for some of our sister companies, and that alone is proving to be a challenge. Trying to meld different corporate cultures is bad enough and then add in the technology layer and it can be really fun! I may cover the culture clashes some other time, but for nowI thought it might be interesting to go over the technology.

We started planning on using Oracle Standard Edition database and Real Application Clusters running on Oracle Enterprise Linux using HP hardware with Intel x86-64 processors. Many of our discussions centered around the version of Oracle and whether or not to implement RAC. We made the decision to stick with Standard Edition even though all of our current implementations are Enterprise Edition because of budgetary concerns. The plan is to move to Enterprise Edition in the future so we can standardize our licensing, but for now, we have a limited budget and are implementing Standard Edition. We also decided to forgo RAC and user Clusterware to protect the databases using active/passive clustering instead of active/active. Our timeline was very short and I have used Clusterware in this fashion before so this helped to shorten the timeline, but we also don't really know how much traffic we are talking about so RAC may not even be necessary.

So we have Clusterware installed and running, Automatic Storage Management is running and storage is allocated, the RDBMS is installed and patched, and we are ready to create the databases. Next comes the fun part... pushing power buttons and pulling power cords to make sure it all works as planned! This has been an incredible learning experience and should continue to be. Next is building our test environment using Oracle VM! Let me know any comments if you have had similar projects or want to chat about what we're doing.

Until next time... keep learning! Oh, by the way, I'm sure there will be lots of good sessions on Clusterware, RAC, Linux, etc at Collaborate 09 this year.

Thursday, March 12, 2009

Oracle Magazine Interview

Today was kind of an exciting day for me. I was interviewed via webcam and Skype for Oracle Magazine. They do a Linux issue every year and they have started doing a feature on the user community. I am the current president of the Linux Special Interest Group (SIG) for the Independent Oracle User Group and they asked if they could interview me about the SIG and our relationship with Oracle. I was contacted by Jeff Erickson at Oracle Magazine through Rich Schwerin, who I mentioned in an earlier blog post. Rich is the Senior Product Marketing Manager in the Linux and VM group at Oracle. We talked for about a half hour about what the SIG has done in the past, our future plans, and how Oracle is doing in the Linux and support areas. I won’t ruin it for you and you’ll have to check back to find the link to the video, but I really appreciate the opportunity to talk about the IOUG and the Linux SIG.

For part of the interview we discussed personal and professional opportunities that come from involvement, not just membership, but involvement, in the user group community. Personally, I have made some great friendships and have people that I talk to, either on the phone or electronically, regularly. Professionally I have made some networking relationships that have saved me several times. This is where involvement really comes in. I started with IOUG as a member attending the conferences. That was great, and I learned a lot, but it was always lacking a little something for me. Once I started actually getting involved and taking a leadership role in the SIGs, the experience changed. The interview also covered my experience with Linux and the relationship with Oracle. At my previous company, we implemented Oracle Clusterware on Red Hat Linux. Those networking relationships really came in handy there! Now, my coworkers and I are embarking on a Clusterware project using Oracle Enterprise Linux. I'm looking forward to getting back into it as well as having something to share with the Linux SIG.

So, I will post the link to the video in a couple months. Hopefully I didn’t look too geeky. In the meantime, make sure you are getting the most out of your career by getting involved. And don't forget that Collaborate 09 is coming up in May. There will be several sessions on Linux and over 350 sessions total just from the IOUG Forum. IOUG is always looking for good volunteers that are willing to step up and take a leadership role. The SIGs are a very good way to do that. Who knows… maybe next time you could be the one being interviewed for the magazine! Speaking of that, make sure you check out IOUG President Ian Abramson’s blog today. He is also talking about being interviewed by the media. He also has a good post about some of the inner workings of the IOUG Board from yesterday.

Monday, March 9, 2009

Collaborate 09 Paper: PeopleSoft Management Plug-in for Oracle Enterprise Manager (aka Best Laid Plans)

“The best laid plans of mice and men often go awry.”

I’m working on writing my paper for Collaborate 09, and for some reason, this phrase keeps coming back to me. I actually looked up this quote and it is often misquoted, including in my opening sentence. The quote is actually from a poem by Robert Burns, called “To a Mouse” and the quote is actually, “The best laid schemes o’ mice an’ men, Gang aft a-gley.” Burns, incidentally, also wrote Auld Lang Syne. My best laid plans included actually having my white paper and presentation completed for Collaborate 09 before they were due. This year my presentation is “Implementing the PeopleSoft Enterprise Environment Management Plug-in for Oracle Enterprise Manager.” I did get an early start, but have been working on this for about six months and have been unable to complete the project.

Issues started almost immediately with the installation process. We are an HP-UX shop running on Itanium CPUs. The installation of the plug-in to the Oracle Enterprise Manager Management Service failed with a password issue. After several weeks, Oracle determined that the installer was having issues on Itanium. They created a patch for the installer that allowed me to install the plug-in and I was finally on my way. You will have to wait for my paper and presentation for the details of the installation and setup, but I did run into a few more road blocks that fed into the “Best Laid Plans” theme.

One thing that got me was that I had installed the Oracle Management Agent on my PeopleSoft servers before starting to do the research for the PeopleSoft plug-in. Big mistake! I was hoping to monitor the up/down status of the hardware and the available space for the file systems. This worked well, but then I found out that the installation of the agent for the plug-in is required to be owned by, or at least in the same group as, the user that owns the PeopleSoft installation. So after several uninstalls and reinstalls, I was ready to go again. Of course then there was the issue of what user to use when discovering the PeopleSoft targets. The documentation is unclear and I tried several options. Bottom-line is that while the Management Agent can run as a centralized user that shares the group with the PeopleSoft installation, the discovery has to be done by the actual software owner. This was an issue only because we have several environments hosted on the same server in our non-production environments. That was a long paragraph and hopefully makes some sense. If not, I wrote it much better in the paper so you can check it out at Collaborate.

My next challenge had to do with how PeopleSoft identifies its databases. This is something I had no clue about and caught me off-guard. I went through and, with the help of the PeopleSoft Administrators, added all the non-production PeopleSoft targets. One of the cool features of the plug-in is that it allows you to create a system for each environment and a service to measure the login time and make sure the web site is available. When I started to create systems for each environment, Enterprise Manager tried to connect all the components to one database. After further research I found that PeopleSoft stores a GUID in the PSOPTIONS table that needs to be unique for every environment. Since we refresh our test environments with production, all the GUID fields were the same, thus the reason that Enterprise Manager tied everything together. The fix for that is to shut down the application server, set the GUID to a single space in PSOPTIONS, and then restart the application server. The application server then assigns a new GUID and all is wonderful… Unfortunately, the developers, testers, and PeopleSoft administrators are not that happy to give up the environments for outages. Plus, I had to delete all the targets that had been previously discovered and go through the discovery process all over.

At this point I am almost ready to add the production targets to the Enterprise Manager repository and really start using it like it is meant to be used. And of course, I found one more issue. We refreshed one of our non-production environments the other day. Part of the service monitoring includes a user in the PeopleSoft system that only has rights to log in and nothing else. Of course, we decided to use a different password that we could standardize for all non-prod environments, but it is different than what we used the first time. I cannot for the life of me, figure out where to change that password to make the service check work again. I ended up creating a Service Request with Oracle and apparently, they don’t know either. It has been a week and they haven’t given me a response.

Well, I better actually get to finishing the paper. I have not been able to find a lot of information about people using the plug-in so if you are, please drop me a line and let me know. I would love to trade war stories! Also, if you are not already planning on attending Collaborate 09 in Orlando in May, make sure you check it out. It is hard to beat the amount of technical content and the networking opportunities that exist at this one conference.

Until next time, and as always… stay clean!

Wednesday, March 4, 2009

Upcoming Webinar – Auditing Database Activity for Security and Compliance with Oracle Audit Vault

In my initial entry, I mentioned that I am involved with the Independent Oracle User Group and the Special Interest Groups. The Enterprise Best Practices SIG is sponsoring a very exciting webinar next week that any Oracle technologist should be participating in. Tanya Baccam is an instructor with the SANS Institute and Roxana Bardescu is a Senior Principal Product Director in the Database Security Marketing Group at Oracle. They are co-presenting a webinar on Audit Vault, Oracle’s new tool for capturing audit information. The webinar is scheduled for Wednesday, March 11 at 2:00 pm Central Time. Here is the abstract:

Using the native auditing capabilities provided by the Oracle Database and Oracle Audit Vault, organizations can easily monitor their databases for suspicious activities and simplify compliance reporting without costly investments in third-party solutions. In this session, you will learn best practices for effectively auditing your Oracle databases as well as how Oracle Audit Vault can proactively collect and analyze that data in real-time to generate alerts and reports that satisfy internal security policies and regulatory compliance.

I have heard both Roxana and Tanya present before and they are very knowledgeable. This should be a very informative webinar. You can register by following this link.

If you want more on security, you can really get an in depth view of database security at Collaborate 09. Tanya will be presenting a University Seminar on Sunday that is an abbreviate version of the SANS Institute’s week long Securing Oracle Database training. This will give you enough information to scare the heck out of your manager!

Check out the webinar… nothing like free training in this day! Happy auditing!

Oracle Releases Grid Control 10g Release 5 – Follow-up

I just received a follow-up to the webinar yesterday and it is now posted online. You can go to Oracle Technology Network and listen to the recorded webinar here.

After my post yesterday, I received a note on LinkedIn from Rich Schwerin, Senior Product Marketing Manager with the Linux and VM group at Oracle. I have worked with Rich in the past on a roundtable at OpenWorld 2008 and he put me in touch with a couple people for webinars for the IOUG Linux SIG. He reminded me about the new VM features that Oracle put into 10gR5. This is very cool because you can manage the entire stack, right down to the virtual hosts, through the centralized console. Add that to the Linux management pack that is included with your support license for Oracle Enterprise Linux and you really do have a top down management opportunity.

I encourage you to check out the webinar and technical information if you are a VM and/or Enterprise Linux user. The new features should help to make our jobs much easier.

Thanks for the note Rich!

Tuesday, March 3, 2009

Oracle Releases Grid Control 10g Release 5!

I know I said my next entry would be about my paper for Collaborate 09, but I sat through Oracle’s webinar today about Grid Control 10gR5 and decided to write about that instead. I’ve been waiting for this release because a lot of the 11g database features have been built into the Enterprise Manager interface. Real Application Testing and Transparent Data Encryption are features that are in the 11g database, but you had to use the local Enterprise Manager to use them. Now they are built into Grid Control. The release number is 10g Release 5, but this was actually supposed to have been the 11g release. Rumor has it, and presenters at conferences have backed this up, that they were ready to call this 11g, but when the BEA purchase went through they decided to hold off on the 11g release until they evaluate revamping Grid Control to use WebLogic instead of Oracle Application Server. We shall see…

The webinar from today is supposed to be posted by tomorrow, but the slides are already there. This really does look like a very nice, feature packed, upgrade. We are on HP-UX/Intanium at my company so I have to wait a bit longer, but if you are using Windows 32 or Linux/Intel, you are good to go. Check out the Enterprise Manager page on OTN and read about the new features. The Real User Experience Insight looks to be a nice feature for measuring performance at the user level. There are accelerators already created for E-Business Suite, Seibel, and PeopleSoft. This should be fun to play with since I support PeopleSoft.

Oracle is building in their acquisitions as fast as they can. One new acquisition that was mentioned on the webinar is mValent. The acquisition was just completed a couple weeks ago and Oracle is already working to build those features into Grid Control. I’m not familiar with the mValent software at all, but apparently it is supposed to make managing complex applications even easier. It should be interesting to see how all these new features for managing complex application environments fit in with already existing plug-ins. As you will see in my later post, I have had several headaches getting the PeopleSoft plug-in implemented and if these tools turn out to be easier, it may be the end of the current plug-in model.

I better get going on my Collaborate paper. Hopefully once I finish it, I can actually write the blog posting about it. Feel free to drop me a line if you like the blog or have some suggestions. Also, I’d love to hear from you if you have had a chance to play with 10gR5!