Wealth of Oracle Knowledge

I have been working with Oracle technologies for almost 15 years. One thing that has changed over that time is how we gain knowledge about the technologies. I was fortunate when I graduated from college and my company sent me to several weeks of training. I learned a new programming language and took all the Oracle University classes to become certified in Oracle 7 (yes, Oracle 7).

Today, Oracle University is still going strong, but we also have Google, blogs, Twitter, and, of course, User Groups.  With this wealth of knowledge we can find just about anything at any time. The question then becomes, is this the best answer, or just an answer.  This is where I feel that the User Groups set ourselves apart.  Not only is the information there, but it is being written, presented, and discussed by the thought leaders. IOUG is participating in three events that really underline this point.

As I write today, I am attending the International Oracle User Community User Group Leaders' Summit. This is an event that is sponsored by Oracle and takes place on the Oracle campus in Redwood Shores. This meeting is a mini conference for leaders of the Oracle Community. There is some great information shared by Oracle here, but the bigger reason to attend is to talk with other leaders to find out what they are doing and what has been successful. There is a large focus this year on MySQL and Java given the Sun acquisition that was finalized at this time last year.  We are all still learning from each other and it is exciting to get to know two entirely new user communities. I will be taking a lot of this information back to IOUG and working on growing our community, but I will also take some of the new information back to my day job.


The next, very exciting, event that we are involved in was just recently announced.  This new event is called "IOUG Presents A Day of Real World Performance with Tom Kyte, Andrew Holdsworth and Graham Wood". This is going to be a full day of performance engineering with the three biggest experts within Oracle. These Oracle experts will debate, discuss and delineate the best practices for designing hardware architectures, deploying Oracle databases, and developing applications that deliver the fastest possible performance for your business. This is going to be an excellent way to learn directly from the experts and have an opportunity to ask direct questions. There are four sessions currently scheduled on the West Coast, but we are working on scheduling more sessions throughout the country.  Check out the information Real World Performance Tuning page at ioug.org to see the currently scheduled dates and check back soon for future dates.  Also, let me know if you are interested helping with this event in your area and I will get you in touch with our scheduling people.


Finally, our annual conference planning is in full swing for Collaborate 11. IOUG, OAUG and Quest International user communities come together annually to put on the on of the leading, user based, Oracle technology conference in the United States. I am currently working on two papers, including one that will be part of the Oracle RAC Boot Camp. Going back to our roots, you will find a DBA 101 Boot Camp along with BI, Development, Manageability and many others.  The other exciting news from Collaborate this year is MySQL. With Oracle's acquisition of Sun last year, the MySQL Community is now part of the Oracle Community. IOUG is offering over 50 sessions just on MySQL. Those of you that are multi database shops can now come to Collaborate and get all your answers. I am excited about being a part of Collaborate again this year. We are going to have our best show ever.


The User Community is busier than ever with more opportunities to learn and get involved. Join us for a Real World Performance session or at Collaborate. I look forward to seeing a lot of people that I see once a year and meeting many more.

Oracle Technology Day - Milwaukee

Today I am participating in Oracle Technology Day in Brookfield, WI.  Brookfield is a suburb of Milwaukee and closer to my house than work so the commute was better than normal.  Those of you that participate in these regularly know that it is an opportunity for Oracle to buy us breakfast and lunch, and in return, they get an opportunity to tell us about their technology.  We also get an opportunity to network with several people from the area and find out what others are doing.  Meeting others from the are is the best part of these days.  The technology discussions are interesting, but most of it is somewhat of a repeat from OpenWorld.  So far I have seen about ten people that I have met somewhere in my career history.  This is also why I am involved in IOUG because I get this kind of interaction regularly.

Right now is an update on Oracle Technology keynote.  We are currently going over the history of the database and leading into where we are now.  This is an interesting story to think of going from Oracle 4 back in 1984 to 11gR2 running in a Grid environment now.  Add to that where we are going with Cloud Computing and it makes for a very interesting story.  Today includes information on Exadata, Security, Fusion Middleware, BI, GoldenGate and much more.  I will be jumping around among the three tracks because it all is very interesting and we have things going on in all areas at work and at IOUG.  I like to use these days to find out more about what we can be getting to our members.  It's funny how all the topics that are covered today are the same topics we hear that our membership is interested in.  That is partially due to the User Groups giving Oracle feedback about what our members want.  Oracle has done a good job recently with allowing the User Groups to have advocacy back into Oracle and allow us to get the message out.  Of course, we still like to talk about what doesn't work and what the products really do which is why we stay independent.

I'm looking forward to hearing more about Fusion Middleware and BI opportunities.  Also talking about IT Sprawl and better management of the data center should be good topics.  These will be helpful as we grow our current RAC environment and start to consolidate databases onto it.  I probably will skip some of the Exadata stuff since Larry talked so much about it at OpenWorld and my current company really doesn't have the volume to warrant that kind of investment.  It still sounds very cool though!  Again, I'm mostly looking forward to lunch and being able to catch up with people I know and meeting some new ones.  Everyone has a different technology story and you never know when you might pick up something you can use.  I'm already looking forward to April and Collaborate.  That story sharing is a daily thing at Collaborate.  Registration for Collaborate opens next month.  Make sure to check it out and come share your story.  Watch for updates at http://collaborate11.ioug.org.

Hopefully, I'll have some new information to share after Tech day today.  I'm sure I will at least have some new contacts and stories!

OpenWorld User Group Forum - Best Day of the Week! #oow10

It has been an exciting and fast paced start to the week.  At the moment I am in the User Group Pavilion talking to a lot of people about the benefits of joining a users group.  I've actually been trying to get to the post for a couple hours and get a few words in before saying hi to someone.

If you are here, stop by the booth and check us out.  I'd love to say hi and learn about what you are doing here. If not, check out www.ioug.org and find out what we are up to.  Especially remember about Collaborate 11 in April.  This afternoon you can get a sneak peak of the content that you will see at Collaborate during the User Group Forum.  All the IOUG sessions are in the 2001-2008 block of rooms in Moscone West.

More later... we have a line!

Error Stack

Most people think of an error stack as the list of errors that is generated when there is an issue.  For example and ORA-00600 causes a few other errors and gives you a stack of errors to unravel.  In our case, earlier this week we had a stack of errors that caused a minor database outage.  We went live a few weeks ago with Oracle Real Application Clusters to support PeopleSoft Financials.  This was a major upgrade from our standalone HP-UX server.  Of course there were several things that we learned before going live, but a couple caught us by surprise.

With the previous architecture, we were using Data Guard and replicating to a hot disaster recovery site.  Shortly after go-live with RAC, we were working on getting this set up again.  I only go into this part because it was one of the things that caused our error stack.  In short, the "error stack" is below.  I will explain more shortly.

"Error Stack"

1.  Production database lost contact with DR
2.  No monitoring of DR site being up to date
3.  RMAN backups successful, but not deleting logs because they were needed for DR
4.  Archive log destination filling up
5.  Monitoring was set, but no e-mail or page notification

Error number one that would have avoided our production outage was the logs not reaching the Data Guard server.  We are still unsure of why the production site lost contact with DR.  The current theory was an incomplete setup of Data Guard Broker.  We were received TNS errors in the alert log, but these were not reported in OEM.

Before we went live with the RAC environment, we had Data Guard running for our single instance 10g database and had a script that checked the sequence number between production and DR to make sure we were up to date and would page if we got too far behind.  With the implementation of RAC, that script no longer worked and we did not get a chance to fix it when we turned on the Data Guard database.  That was error number two that if we had caught, we would not have had an outage.

Our RMAN backups are scripted and we use OEM pretty much only as a job scheduler.  This was setup originally because the company did not have OEM up and running, and when we did get it implemented, we didn't want to rely on it for backup/recovery because we did not have a DR plan that included OEM.  That way we could just run the scripts for backup/recovery in the event of a DR or an OEM outage.  The logs from the scripts are e-mailed to the DBA's, but unless there is a real problem, they come up as successful.  In this case, the backups of the archive logs were successful, but our scripts are supposed to delete the logs and that part was failing.  We did not know the delete was failing because the overall script and the backup itself was successful.  RMAN was not deleting the logs because they could not be transmitted to the DR site and RMAN was trying to protect the DR site.

Since the archive logs were not being deleted as we expected by RMAN, our archive log destination filled up.    We still have to figure out what is different between 10g and 11g here and I'm assuming for now that this is a new feature since we didn't really change anything else.  Since, we changed our scripts to do force delete on the archive logs because we can always restore them if we need them for DR.

The monitoring piece of the error stack is the one that bugs me the most.  I had set the monitoring up via OEM and the alert was actually triggered.  The kicker was that I forgot to set the e-mail and pager notifications.  If we had happened to be looking at the alert screen on OEM we would have seen it.  Obviously, this has been remedied and we are not receiving e-mail messages and pages.

So, there you go.  Our version of an error stack.  When unraveling all of this, I kept thinking to myself that any one of these would have saved us an outage.  Obviously a learning experience, but still frustrating to know we missed that many little things that added up to a big one.  Fortunately, we did get notified that the database was unavailable and the total outage was only about 15 minutes.  I guess it can always be worse!

Hope to see many people at OpenWorld.  One of our PeopleSoft administrators and I are presenting the implementation of our PeopleSoft upgrade and RAC implementation so if you want to chat look me up.  I will be posting something more soon about other plans for OpenWorld.

My OpenWorld 2010 Planning

I am supposed to be working on the next post for my PeopleSoft/RAC project, but decided to take a break from that and talk about OpenWorld.  As just about anyone in the Oracle universe is aware, OpenWorld is September 19th to 23rd.  This year I am doing three sessions along with helping to organize the User Group Forum on Sunday.  Before that all gets going though, I'm hoping to catch the Milwaukee Brewers at the San Francisco Giants.  I am a huge Brewers fan and I have never seen a game outside of Milwaukee.  The Brewers are in San Francisco for a weekend series and I'm hoping to get to the game on Saturday evening.  Anyone want to join me?

Sunday is again User Group Forum day at OpenWorld.  The schedule is a little different and the sessions are not starting until 12:30 do to travel restrictions on Satruday because of Yom Kippur.  IOUG again has eight rooms that we have schedule four sessions per room.  In addition, Oracle is having MySQL Sunday during the same time at the Marriott.  IOUG will be participating with some sessions there too.  This will split the audience a little bit on Sunday, but we are hoping for huge crowds at both events.  Check out the IOUG list of sessions or via Schedule Builder or Content Catalog on the OpenWorld site.

My personal schedule is already busy as usual.  I will be helping out at the IOUG booth again along with various meetings.  Plus, I am involved with three sessions this year.  For the User Group Forum on Sunday, I will be leading a round table discussion about Linux.  We will have several users as well as people from the Oracle development team.  This has been a very interesting and well attended session in years past.  People have had many questions and there is a lot of information sharing that takes place.  If you are an Oracle user that has anything installed on Linux, this should be a very informational sessions.  Come and learn about Linux and also share your experiences.

Recently, my company updated our PeopleSoft application.  As our PeopleSoft Administrator (PSA) likes to say, "we did a minor upgrade."  That "minor" upgrade included PeopleTools 8.49 to 8.50.08, Application Bundles from 25 through 28, and implementing Multi-Language.  We are toward the end of phase 2 of the project, which is to move to Oracle Real Application Clusters (RAC) for the database.  The PSA that was in charge of phase 1 and I are doing a presentation on the overall project.  This should be fun because we should be crossing a large number of functions and users.  The title is more about PeopleTools 8.50 which is fine because I think it will draw a big crowd.

When we decided to replace our PeopleSoft database server we went through an evaluation process to determine the best solution.  As part of that evaluation, we decided to use Linux as our database operating system.  I have been a Linux person for several years and had to convert our HP-UX system administrator into an accepter of Linux.  As part of that conversion, he and I will be doing a Linux session with the Linux product team from Oracle.  They will lead in with how the support model is supposed to work and what is included.  We will finish up with how it all really works.

That's my current wrap up for OpenWorld.  I'm hoping to see a lot of people that I only get to see a couple times a year.  I also get to meet new people every year and it blows my mind how many people are out there that are doing some really cool stuff.  If you are planning on attending, let me know and check out the User Group Forum on Sunday!

Oracle RAC For PeopleSoft - Part 1

So it's been several months since my last blog entry.  Things have been crazy at work, at home, and with IOUG and unfortunately, blogging is somewhat low on the priority list.  So, what have I been up to?  Well, the home life  would be enough for its own blog with two kids that are very involved in sports.  That is another story though.  Work has been very interesting with a couple projects.  The one I am in the middle of now is implementing Oracle Real Application Clusters (RAC) to support our PeopleSoft database.

I have been a DBA for about 13 years.  It is almost crazy to say that, but okay, I accept it.  I am on my second time around supporting a PeopleSoft environment.  I work for a company that is made up of many divisions and is part of a larger organization made up of several companies.  We have been consolidating financial systems into a PeopleSoft 9.0 Financials system from our divisions, and recently, from one of our sister companies.  Not only is it a challenge to integrate business processes between divisions and companies, but technically, we have been challenged to find a solution that will grow as we continue to consolidate.  The biggest part of that challenge is that we are not sure, from an IT perspective, when we will be consolidating.  We do not want to oversize our architecture now for where we are, but do not want to  undersize for future growth.

Performance of the PeopleSoft database has been questionable since I started at this company two years ago.  We have done quite a bit with indexes, changing some business processes, and actually changing some of the application processes.  A couple big wins involved converting PeopleSoft "temp" tables to Oracle temporary tables.  Even with all the tuning successes, adding many users through growth and consolidation has made it very apparent that we have outgrown our database environment.  This is really not bad considering the environment was implemented over three years ago with limited information.

Given that we had exhausted most of our tuning options and we knew we were bringing on new projects and users, we decided that new hardware was required.  The decision came down to a new, dedicated database server or Real Application Clusters.  The company is still working to consolidate IT operations, so we are not sure what divisions or how many users we will be supporting over the next three to five years.  Since we did not want to suggest spending several hundred thousand dollars on a new server this year, only to have to do it again in a year if we consolidated with another large division, RAC was the logical choice.

This entry has been in the works for several weeks and is getting long enough.  In the coming entries I will talk about how we put together a proof of concept, designed the architecture, built the environment and how the implementation is going.  Stay tuned.  I will make sure the follow-up entries are faster in coming than this one...

Collaborate 10 Less than Two Weeks! #c10

With Collaborate 10 less than two weeks away, I realized I have not written much lately.  This year's show promises to be better than ever and I am really excited for some of the new changes along with the usual networking and learning opportunities.

About a week ago, Oracle announced that the Enterprise Manager 11g Launch Event will be taking place on Thursday during Collaborate.  Working with the Enterprise Manager team and the fantastic IOUG staff, we have secured a room and will be carrying the Launch Event live as a Collaborate session.  The event is scheduled to be from 10:00 until noon on Thursday morning, April 22 Las Vegas time.  This works out perfectly for the final two sessions on Thursday before the closing general session.  The Launch Event should be listed in the IOUG and Joint schedulers soon so make sure to get it on your calendar for the week.

For the IOUG Regional Users Group leaders we will be having two meetings that should give you a new opportunity for networking.  Tuesday morning for breakfast (I know it's early, but at least it's not after the party) we will be having a joint meeting with the Quest International User Group RUGs.  This will be an opportunity for the RUG leaders from both organizations to meet and discuss what has worked and not worked for the groups.  Also, we are hoping this might promote some co-location of meetings.  The other opportunity is for the RUG leaders to meet with Oracle field marketing and get to know some of the points of contact in their area.  Dan Goldstein, Senior Director for Oracle Field Marketing, will have his team at Collaborate.  We have set up a lunch meeting for th RUG leaders to meet with the point of contact in the area.  If you area  RUG leader, make sure to check these out and make use of the networking opportunities.

The IOUG Special Interest Groups are also taking a large role again this year.  Many of the SIGs are having annual meetings and round table discussions.  Some of the SIG leaders also participated in creating the content for the Boot Camps, but more on those in a bit.  If you are interested in the user group experience and want to get involved, the SIGs are a great place to start.  Find a topic you are interested in, look up that SIG meeting and join in.  You have nothing to lose and so much to gain.  Enterprise Best Practices, Linux, RAC, BI, Content Management and more are all available.  New this year is the zLinux SIG.  This group used to be the z/OS SIG and have joined forces with the IOUG.  The are also having their own boot camp at Collaborate 10.

Several other new options should prove to make the conference experience even better this year.  If you can't make it for the entire week, check out the two day registration option.  This allows you to take advantage of two days of educational content and pay less than half the cost of the entire conference.  Another option this year is the introduction of Boot Camp style sessions.  Several topics were chosen and all the content for that topic was centralized into a one or two day schedule.  Need to learn about RAC?  Check out the sessions in the two day RAC Boot Camp.  Most sessions are in the same room, so you can just camp out there and learn all you need to know.

Business Intelligence has become one of the hottest areas of IT in a long time.  Get Analytical with BIWA Training Days is our conference within a conference for all things BI.  Anything you wanted to know, and some things you didn't want to know, are all in one location.

Add all of this to what Quest and OAUG are offering and this is the most complete conference around for Oracle technology by users.  Check it out and register here.... http://collaborate10.ioug.org

I hope to see you in Vegas!  If you are there, make sure to look me up and say hi!