Cheesehead DBA

IOUG Real World Performance - Milwaukee

2011-09-06T09:20:00.000-07:00

The IOUG Real World Performance Tour has finally arrived in Milwaukee! I have felt a little like a teenager waiting for my favorite band to come to town. Tom Kyte, Andrew Holdsworth and Graham Wood are here talking about Oracle performance and everyone in the room is listening with rapt attention. So far, we know that most performance problems are the network guys... something we DBA's have known for a long time. In reality, the performance issues come in all over the stack and limiting those become our job. Andrew just loaded 2.5 million rows in a minute and a half. I have already learned a lot about SQL Loader, external tables and statistics.

Much of this information is stuff that Tom, Andrew and Graham have learned over many years of working with the product. There are things here that have been learned through Exadata development and are still valid in the non-Exadata world.

I would highly recommend checking out this program if it is coming to a town near you. If it isn't, and you are interested in having the tour visit your area, let the IOUG know!

Recommendation Pays Off (aka I Told You So!)

2011-03-23T12:01:00.000-07:00

Yesterday, Oracle announced that they will no longer be developing for the Itanium processor. SearchOracle posted a blog this morning with a few more details. This caught my attention because last summer my company made the move from HP-UX on Itanium to Oracle Enterprise Linux on X86-64. The initial plan a year ago was to replace the current Itanium server with a bigger one to support our growing processing needs of the Oracle environment. When we started scoping the project, the DBA team suggested looking at Oracle RAC and using smaller, commodity hardware with Linux. Long story short, that was the ultimate decision, and based on the news yesterday, turned out to be the right one. Any time you can make a recommendation that ultimately saves the company time and money, you can call that a success. In this case, if we had moved forward with replacing the existing platform with the same processor, we would be looking for another replacement.

Understanding the technology that you are supporting in your job is just the start. This example shows that keeping an eye on trends and really trying to understand where your company is and where you are going is even more important than being a good technologist. Much of the decision came down to the cost of Itanium, but knowing that the technology did not have the backing that it once did had an impact. Also, Oracle had already put Itanium into its second round anytime releases came out. I was able to learn about those trends while being involved in the Oracle user community. A big part of that is attending conferences like Oracle's OpenWorld and Collaborate which is co-hosted by IOUG, OAUG and Quest. Collaborate 11 will be April 10-14 in Orlando. This is a great way to learn about the latest trends in the industry and learn the latest technology from experts that do it every day. I will be presenting my paper about moving from Itanium to Oracle RAC on Linux. Register today!

Wealth of Oracle Knowledge

2011-01-25T14:28:00.000-08:00

I have been working with Oracle technologies for almost 15 years. One thing that has changed over that time is how we gain knowledge about the technologies. I was fortunate when I graduated from college and my company sent me to several weeks of training. I learned a new programming language and took all the Oracle University classes to become certified in Oracle 7 (yes, Oracle 7).

Today, Oracle University is still going strong, but we also have Google, blogs, Twitter, and, of course, User Groups. With this wealth of knowledge we can find just about anything at any time. The question then becomes, is this the best answer, or just an answer. This is where I feel that the User Groups set ourselves apart. Not only is the information there, but it is being written, presented, and discussed by the thought leaders. IOUG is participating in three events that really underline this point.

As I write today, I am attending the International Oracle User Community User Group Leaders' Summit. This is an event that is sponsored by Oracle and takes place on the Oracle campus in Redwood Shores. This meeting is a mini conference for leaders of the Oracle Community. There is some great information shared by Oracle here, but the bigger reason to attend is to talk with other leaders to find out what they are doing and what has been successful. There is a large focus this year on MySQL and Java given the Sun acquisition that was finalized at this time last year. We are all still learning from each other and it is exciting to get to know two entirely new user communities. I will be taking a lot of this information back to IOUG and working on growing our community, but I will also take some of the new information back to my day job.

The next, very exciting, event that we are involved in was just recently announced. This new event is called "IOUG Presents A Day of Real World Performance with Tom Kyte, Andrew Holdsworth and Graham Wood". This is going to be a full day of performance engineering with the three biggest experts within Oracle. These Oracle experts will debate, discuss and delineate the best practices for designing hardware architectures, deploying Oracle databases, and developing applications that deliver the fastest possible performance for your business. This is going to be an excellent way to learn directly from the experts and have an opportunity to ask direct questions. There are four sessions currently scheduled on the West Coast, but we are working on scheduling more sessions throughout the country. Check out the information Real World Performance Tuning page at ioug.org to see the currently scheduled dates and check back soon for future dates. Also, let me know if you are interested helping with this event in your area and I will get you in touch with our scheduling people.

Finally, our annual conference planning is in full swing for Collaborate 11. IOUG, OAUG and Quest International user communities come together annually to put on the on of the leading, user based, Oracle technology conference in the United States. I am currently working on two papers, including one that will be part of the Oracle RAC Boot Camp. Going back to our roots, you will find a DBA 101 Boot Camp along with BI, Development, Manageability and many others. The other exciting news from Collaborate this year is MySQL. With Oracle's acquisition of Sun last year, the MySQL Community is now part of the Oracle Community. IOUG is offering over 50 sessions just on MySQL. Those of you that are multi database shops can now come to Collaborate and get all your answers. I am excited about being a part of Collaborate again this year. We are going to have our best show ever.

The User Community is busier than ever with more opportunities to learn and get involved. Join us for a Real World Performance session or at Collaborate. I look forward to seeing a lot of people that I see once a year and meeting many more.

Oracle Technology Day - Milwaukee

2010-10-28T07:27:00.000-07:00

Today I am participating in Oracle Technology Day in Brookfield, WI. Brookfield is a suburb of Milwaukee and closer to my house than work so the commute was better than normal. Those of you that participate in these regularly know that it is an opportunity for Oracle to buy us breakfast and lunch, and in return, they get an opportunity to tell us about their technology. We also get an opportunity to network with several people from the area and find out what others are doing. Meeting others from the are is the best part of these days. The technology discussions are interesting, but most of it is somewhat of a repeat from OpenWorld. So far I have seen about ten people that I have met somewhere in my career history. This is also why I am involved in IOUG because I get this kind of interaction regularly.

Right now is an update on Oracle Technology keynote. We are currently going over the history of the database and leading into where we are now. This is an interesting story to think of going from Oracle 4 back in 1984 to 11gR2 running in a Grid environment now. Add to that where we are going with Cloud Computing and it makes for a very interesting story. Today includes information on Exadata, Security, Fusion Middleware, BI, GoldenGate and much more. I will be jumping around among the three tracks because it all is very interesting and we have things going on in all areas at work and at IOUG. I like to use these days to find out more about what we can be getting to our members. It's funny how all the topics that are covered today are the same topics we hear that our membership is interested in. That is partially due to the User Groups giving Oracle feedback about what our members want. Oracle has done a good job recently with allowing the User Groups to have advocacy back into Oracle and allow us to get the message out. Of course, we still like to talk about what doesn't work and what the products really do which is why we stay independent.

I'm looking forward to hearing more about Fusion Middleware and BI opportunities. Also talking about IT Sprawl and better management of the data center should be good topics. These will be helpful as we grow our current RAC environment and start to consolidate databases onto it. I probably will skip some of the Exadata stuff since Larry talked so much about it at OpenWorld and my current company really doesn't have the volume to warrant that kind of investment. It still sounds very cool though! Again, I'm mostly looking forward to lunch and being able to catch up with people I know and meeting some new ones. Everyone has a different technology story and you never know when you might pick up something you can use. I'm already looking forward to April and Collaborate. That story sharing is a daily thing at Collaborate. Registration for Collaborate opens next month. Make sure to check it out and come share your story. Watch for updates at http://collaborate11.ioug.org.

Hopefully, I'll have some new information to share after Tech day today. I'm sure I will at least have some new contacts and stories!

OpenWorld User Group Forum - Best Day of the Week! #oow10

2010-09-19T11:27:00.000-07:00

It has been an exciting and fast paced start to the week. At the moment I am in the User Group Pavilion talking to a lot of people about the benefits of joining a users group. I've actually been trying to get to the post for a couple hours and get a few words in before saying hi to someone.

If you are here, stop by the booth and check us out. I'd love to say hi and learn about what you are doing here. If not, check out www.ioug.org and find out what we are up to. Especially remember about Collaborate 11 in April. This afternoon you can get a sneak peak of the content that you will see at Collaborate during the User Group Forum. All the IOUG sessions are in the 2001-2008 block of rooms in Moscone West.

More later... we have a line!

Error Stack

2010-09-09T19:02:00.000-07:00

Most people think of an error stack as the list of errors that is generated when there is an issue. For example and ORA-00600 causes a few other errors and gives you a stack of errors to unravel. In our case, earlier this week we had a stack of errors that caused a minor database outage. We went live a few weeks ago with Oracle Real Application Clusters to support PeopleSoft Financials. This was a major upgrade from our standalone HP-UX server. Of course there were several things that we learned before going live, but a couple caught us by surprise.

With the previous architecture, we were using Data Guard and replicating to a hot disaster recovery site. Shortly after go-live with RAC, we were working on getting this set up again. I only go into this part because it was one of the things that caused our error stack. In short, the "error stack" is below. I will explain more shortly.

"Error Stack"

1. Production database lost contact with DR
2. No monitoring of DR site being up to date
3. RMAN backups successful, but not deleting logs because they were needed for DR
4. Archive log destination filling up
5. Monitoring was set, but no e-mail or page notification

Error number one that would have avoided our production outage was the logs not reaching the Data Guard server. We are still unsure of why the production site lost contact with DR. The current theory was an incomplete setup of Data Guard Broker. We were received TNS errors in the alert log, but these were not reported in OEM.

Before we went live with the RAC environment, we had Data Guard running for our single instance 10g database and had a script that checked the sequence number between production and DR to make sure we were up to date and would page if we got too far behind. With the implementation of RAC, that script no longer worked and we did not get a chance to fix it when we turned on the Data Guard database. That was error number two that if we had caught, we would not have had an outage.

Our RMAN backups are scripted and we use OEM pretty much only as a job scheduler. This was setup originally because the company did not have OEM up and running, and when we did get it implemented, we didn't want to rely on it for backup/recovery because we did not have a DR plan that included OEM. That way we could just run the scripts for backup/recovery in the event of a DR or an OEM outage. The logs from the scripts are e-mailed to the DBA's, but unless there is a real problem, they come up as successful. In this case, the backups of the archive logs were successful, but our scripts are supposed to delete the logs and that part was failing. We did not know the delete was failing because the overall script and the backup itself was successful. RMAN was not deleting the logs because they could not be transmitted to the DR site and RMAN was trying to protect the DR site.

Since the archive logs were not being deleted as we expected by RMAN, our archive log destination filled up. We still have to figure out what is different between 10g and 11g here and I'm assuming for now that this is a new feature since we didn't really change anything else. Since, we changed our scripts to do force delete on the archive logs because we can always restore them if we need them for DR.

The monitoring piece of the error stack is the one that bugs me the most. I had set the monitoring up via OEM and the alert was actually triggered. The kicker was that I forgot to set the e-mail and pager notifications. If we had happened to be looking at the alert screen on OEM we would have seen it. Obviously, this has been remedied and we are not receiving e-mail messages and pages.

So, there you go. Our version of an error stack. When unraveling all of this, I kept thinking to myself that any one of these would have saved us an outage. Obviously a learning experience, but still frustrating to know we missed that many little things that added up to a big one. Fortunately, we did get notified that the database was unavailable and the total outage was only about 15 minutes. I guess it can always be worse!

Hope to see many people at OpenWorld. One of our PeopleSoft administrators and I are presenting the implementation of our PeopleSoft upgrade and RAC implementation so if you want to chat look me up. I will be posting something more soon about other plans for OpenWorld.

My OpenWorld 2010 Planning

2010-07-27T12:26:00.000-07:00

I am supposed to be working on the next post for my PeopleSoft/RAC project, but decided to take a break from that and talk about OpenWorld. As just about anyone in the Oracle universe is aware, OpenWorld is September 19th to 23rd. This year I am doing three sessions along with helping to organize the User Group Forum on Sunday. Before that all gets going though, I'm hoping to catch the Milwaukee Brewers at the San Francisco Giants. I am a huge Brewers fan and I have never seen a game outside of Milwaukee. The Brewers are in San Francisco for a weekend series and I'm hoping to get to the game on Saturday evening. Anyone want to join me?

Sunday is again User Group Forum day at OpenWorld. The schedule is a little different and the sessions are not starting until 12:30 do to travel restrictions on Satruday because of Yom Kippur. IOUG again has eight rooms that we have schedule four sessions per room. In addition, Oracle is having MySQL Sunday during the same time at the Marriott. IOUG will be participating with some sessions there too. This will split the audience a little bit on Sunday, but we are hoping for huge crowds at both events. Check out the IOUG list of sessions or via Schedule Builder or Content Catalog on the OpenWorld site.

My personal schedule is already busy as usual. I will be helping out at the IOUG booth again along with various meetings. Plus, I am involved with three sessions this year. For the User Group Forum on Sunday, I will be leading a round table discussion about Linux. We will have several users as well as people from the Oracle development team. This has been a very interesting and well attended session in years past. People have had many questions and there is a lot of information sharing that takes place. If you are an Oracle user that has anything installed on Linux, this should be a very informational sessions. Come and learn about Linux and also share your experiences.

Recently, my company updated our PeopleSoft application. As our PeopleSoft Administrator (PSA) likes to say, "we did a minor upgrade." That "minor" upgrade included PeopleTools 8.49 to 8.50.08, Application Bundles from 25 through 28, and implementing Multi-Language. We are toward the end of phase 2 of the project, which is to move to Oracle Real Application Clusters (RAC) for the database. The PSA that was in charge of phase 1 and I are doing a presentation on the overall project. This should be fun because we should be crossing a large number of functions and users. The title is more about PeopleTools 8.50 which is fine because I think it will draw a big crowd.

When we decided to replace our PeopleSoft database server we went through an evaluation process to determine the best solution. As part of that evaluation, we decided to use Linux as our database operating system. I have been a Linux person for several years and had to convert our HP-UX system administrator into an accepter of Linux. As part of that conversion, he and I will be doing a Linux session with the Linux product team from Oracle. They will lead in with how the support model is supposed to work and what is included. We will finish up with how it all really works.

That's my current wrap up for OpenWorld. I'm hoping to see a lot of people that I only get to see a couple times a year. I also get to meet new people every year and it blows my mind how many people are out there that are doing some really cool stuff. If you are planning on attending, let me know and check out the User Group Forum on Sunday!

Oracle RAC For PeopleSoft - Part 1

2010-07-10T09:27:00.000-07:00

So it's been several months since my last blog entry. Things have been crazy at work, at home, and with IOUG and unfortunately, blogging is somewhat low on the priority list. So, what have I been up to? Well, the home life would be enough for its own blog with two kids that are very involved in sports. That is another story though. Work has been very interesting with a couple projects. The one I am in the middle of now is implementing Oracle Real Application Clusters (RAC) to support our PeopleSoft database.

I have been a DBA for about 13 years. It is almost crazy to say that, but okay, I accept it. I am on my second time around supporting a PeopleSoft environment. I work for a company that is made up of many divisions and is part of a larger organization made up of several companies. We have been consolidating financial systems into a PeopleSoft 9.0 Financials system from our divisions, and recently, from one of our sister companies. Not only is it a challenge to integrate business processes between divisions and companies, but technically, we have been challenged to find a solution that will grow as we continue to consolidate. The biggest part of that challenge is that we are not sure, from an IT perspective, when we will be consolidating. We do not want to oversize our architecture now for where we are, but do not want to undersize for future growth.

Performance of the PeopleSoft database has been questionable since I started at this company two years ago. We have done quite a bit with indexes, changing some business processes, and actually changing some of the application processes. A couple big wins involved converting PeopleSoft "temp" tables to Oracle temporary tables. Even with all the tuning successes, adding many users through growth and consolidation has made it very apparent that we have outgrown our database environment. This is really not bad considering the environment was implemented over three years ago with limited information.

Given that we had exhausted most of our tuning options and we knew we were bringing on new projects and users, we decided that new hardware was required. The decision came down to a new, dedicated database server or Real Application Clusters. The company is still working to consolidate IT operations, so we are not sure what divisions or how many users we will be supporting over the next three to five years. Since we did not want to suggest spending several hundred thousand dollars on a new server this year, only to have to do it again in a year if we consolidated with another large division, RAC was the logical choice.

This entry has been in the works for several weeks and is getting long enough. In the coming entries I will talk about how we put together a proof of concept, designed the architecture, built the environment and how the implementation is going. Stay tuned. I will make sure the follow-up entries are faster in coming than this one...

Collaborate 10 Less than Two Weeks! #c10

2010-04-06T06:05:00.000-07:00

With Collaborate 10 less than two weeks away, I realized I have not written much lately. This year's show promises to be better than ever and I am really excited for some of the new changes along with the usual networking and learning opportunities.

About a week ago, Oracle announced that the Enterprise Manager 11g Launch Event will be taking place on Thursday during Collaborate. Working with the Enterprise Manager team and the fantastic IOUG staff, we have secured a room and will be carrying the Launch Event live as a Collaborate session. The event is scheduled to be from 10:00 until noon on Thursday morning, April 22 Las Vegas time. This works out perfectly for the final two sessions on Thursday before the closing general session. The Launch Event should be listed in the IOUG and Joint schedulers soon so make sure to get it on your calendar for the week.

For the IOUG Regional Users Group leaders we will be having two meetings that should give you a new opportunity for networking. Tuesday morning for breakfast (I know it's early, but at least it's not after the party) we will be having a joint meeting with the Quest International User Group RUGs. This will be an opportunity for the RUG leaders from both organizations to meet and discuss what has worked and not worked for the groups. Also, we are hoping this might promote some co-location of meetings. The other opportunity is for the RUG leaders to meet with Oracle field marketing and get to know some of the points of contact in their area. Dan Goldstein, Senior Director for Oracle Field Marketing, will have his team at Collaborate. We have set up a lunch meeting for th RUG leaders to meet with the point of contact in the area. If you area RUG leader, make sure to check these out and make use of the networking opportunities.

The IOUG Special Interest Groups are also taking a large role again this year. Many of the SIGs are having annual meetings and round table discussions. Some of the SIG leaders also participated in creating the content for the Boot Camps, but more on those in a bit. If you are interested in the user group experience and want to get involved, the SIGs are a great place to start. Find a topic you are interested in, look up that SIG meeting and join in. You have nothing to lose and so much to gain. Enterprise Best Practices, Linux, RAC, BI, Content Management and more are all available. New this year is the zLinux SIG. This group used to be the z/OS SIG and have joined forces with the IOUG. The are also having their own boot camp at Collaborate 10.

Several other new options should prove to make the conference experience even better this year. If you can't make it for the entire week, check out the two day registration option. This allows you to take advantage of two days of educational content and pay less than half the cost of the entire conference. Another option this year is the introduction of Boot Camp style sessions. Several topics were chosen and all the content for that topic was centralized into a one or two day schedule. Need to learn about RAC? Check out the sessions in the two day RAC Boot Camp. Most sessions are in the same room, so you can just camp out there and learn all you need to know.

Business Intelligence has become one of the hottest areas of IT in a long time. Get Analytical with BIWA Training Days is our conference within a conference for all things BI. Anything you wanted to know, and some things you didn't want to know, are all in one location.

Add all of this to what Quest and OAUG are offering and this is the most complete conference around for Oracle technology by users. Check it out and register here.... http://collaborate10.ioug.org

I hope to see you in Vegas! If you are there, make sure to look me up and say hi!

Oracle SQL Developer Book Review

2010-02-10T19:12:00.000-08:00

I have been asked to review the new Oracle SQL Developer 2.1 book by Sue Harper. When I was asked, I had to clarify that I was not a SQL Developer user. I'm an old school DBA that, for the most part, used scripts and SQL Plus. Developers are always bugging me about why I don't use SQL Developer or TOAD or something like that, but I have my scripts and I like my scripts. When I accepted the offer to review the book, it was partially so I could learn more about the tool and see if it could help me do my job better.

The book is published by PACKT Publishing in Birmingham, UK. They sent me an electronic copy of the book to review. If you would like a sneak peak, you can check it out at the publisher's web site here. I actually like the electronic version. It is much easier to carry than an almost 500 page book and makes it easy to switch between SQL Developer and the book. The whole electronic book thing has intrigued me and I have been looking at the Kindle and Nook, but that is information for a different post. The electronic version did come in handy as I started my review on the plane and having my laptop on the tray table took up enough space. I would have had to ask my neighbor to hold the actual book for me if I had a hard copy.

I have started working through the book and when I say working through, I mean working through. Having never actually used SQL Developer before, I am working through all the examples because I would like to learn as I go. I'm sure as I get a little deeper into the book, I will be able to read/review a little faster, but right now, it is kind of fun to go through and play with things. Sue (the author) starts off the book specifically saying that it is not a book to teach SQL, PL/SQL, or Oracle concepts. There is actually a comment that I found was directed at me (even though she doesn't know me). Even though I am a command line person, this book is written to get me on the road to a "GUI life". I like that term and I'm interested in seeing what I can do that may make my day-to-day job easier or faster.

So far, the book has lived up to the introduction. There have been a couple instances that I felt if you don't know Oracle databases, you might be a little confused, but for the most part I have been able to get started pretty quickly. I like the object browsing and editing capabilities of the tool and the introduction to them in the book. The connection handling is also introduced very well. One thing I didn't have when I started was a pre-built database with the example schemas. I was able to create an 11g database on my laptop and included the example schemas. If you are planning on using this book, the example schemas are almost a must. All the examples are based on the example schemas and they have been very nice to have while going through the book.

I will post further updates as I get farther into the book. I'm not much of a PL/SQL developer, but I will do my best as I get into the development chapters. In the meantime, I recommend checking out the sample that is linked above.

User Group Leaders' Summit Day 1 - #oraclesun

2010-01-27T10:26:00.000-08:00

I was fortunate enough to be invited to the Oracle User Group Leaders' Summit at Oracle HQ in Redwood City this week. Once the Oracle-Sun deal was given the go ahead by the EC last week, we were invited to sit in live for the big announcement. My day started early after about five hours of sleep because my flight was late getting in last night. I met some of my fellow IOUG Board members for breakfast and ran into many others including Ronan Miles, UKOUG Chairman, who went to Brussels with IOUG President, Ian Abramson, to give the User Group position on the acquisition. Also ran into ODTUG President, Mike Riley, Quest President, Sue Shaw, and OAUG President, Ray Payne. That sounds like a lot of name dropping, and it is!

From breakfast, we walked over to the Oracle Conference Center and were ushered into the Auditorium. You can check out all my tweets from the session, but to sum it up, they are pushing hard on the integrated solution, complete platform for applications. We heard from Charles Phillips, who hoped we had a less eventful week than he has... Charles also talked about hiring 2000 people and if the current US Administration is looking for a jobs program, "her we are!" I won't go into politics here, but these are the kind of of things that should be focused on and learned from.

John Fowler, who is now EVP of Hardware Technologies at Oracle discussed the investments in Solaris and Sun hardware moving forward. Mike Splain, who seems to be the real techie, outlined the accelerated road maps for UltraSPARC and SPARC64. Oracle is putting more money into these platforms and they are hiring here too! My question continues to be how many people are getting laid off at the same time that they are hiring all these new positions. I'm sure the layoffs will get some money to walk away, but are they really just shifting positions? That hasn't been asked or answered yet.

The key theme continues to be seamless integrations between existing Oracle technologies and the newly acquired server, storage, and operating system. Supposedly all this testing has been done for many of the Oracle applications. Can they get these all working as they are advertising with flash technologies, new storage options, and newer Sun hardware and Solaris all remains to be seen. They keep selling the fact that they are being built and integrated together and if you have issues, you only have one support organization to deal with.

I'm still in the Oracle/Sun session and I'm sure there will be further information later. I also get a tour of the Usability Lab this afternoon which should be pretty interesting. Tomorrow starts the actual meetings including my first OpenWorld 10 planning meeting. I really wish we could finish one conference before starting planning for the next. Speaking of Conferences, don't forget about Collaborate 10! There will be sessions covering some of the new Sun technologies!

Stay Competitive: Join Me at the COLLABORATE 10 – IOUG Forum

2010-01-20T12:01:00.000-08:00

I invite you to join me at COLLABORATE 10 – IOUG Forum, April 18-22, 2010 in Las Vegas, NV. As a member of the IOUG Board of Directors, I am excited to let you know about the educational content that the conference committee has created for the Forum.

You will have the opportunity to attend educational sessions in all of the tracks offered at COLLABORATE, including: Application Strategy and Services, Infrastructure Technologies BI/Data Warehousing/EPM, Database, Development, Governance Risk and Compliance, Master Data Management/Application Integration and Professional Development.

The IOUG Forum does not only offer top-notch educational sessions that cover a variety of topics, but it’s also a great place to network with other Oracle professionals. We are offering many exciting networking opportunities this year. Whether it’s a technical talk during a Birds of a Feather meeting or an informal conversation at the IOUG Welcome Event, the IOUG Forum is the place to meet contacts who can share solutions to your most pressing business questions.

The IOUG will not be the only user group present at the event. For the fifth consecutive year, the Oracle Applications Users Group (OAUG) and Quest International Users Group (Quest) will be co-hosting this event, which means your registration through the IOUG also allows you to attend education sessions offered by these groups.

The main reason I attend is for the networking opportunities with my peers. In all the years I have been attending IOUG events, the opportunity to discuss current technologies and best practices has been the best way I have found to keep myself current. The technical content is also amazing. These are people that are using the technology on a daily basis.

This isn’t some huge vendor show full of sales presentations…it’s presented by users like you and me. If you have been to IOUG events before, you know that we feature real-world education and best practices. There is far too much content to completely list, but you can visit the IOUG Forum Web site to view the most up-to-date education details.

Make sure to register to attend and join me at the IOUG Forum! Please visit the IOUG Web site for further details and to register. Be sure to register by March 18 to save $400 USD off the onsite rate, and choose priority code PC03 during the demographic section (step 4) of the registration process and gain access to a number of IOUG-exclusive offerings. Help me spread the word and send this information to your colleagues and friends who are Oracle users.

Please contact me or the IOUG headquarters at ioug@ioug.org if you have any questions.

I hope to see you in Vegas!

Three Recoveries, Two Refreshes, One Delete

2009-12-30T07:56:00.000-08:00

I feel like it's been a month since my last entry. Oh yeah, that's because it has been. I continue to be amazed with how fast time flies the older I get. Kids are supposed to make you feel younger and as they grow up, I just feel older!

The title of this post describes a full day I had a few weeks ago with a vendor application. Another title could have been "Support That Cannot Be Reached is No Support At All." The application in question has a relatively small database, but a couple hundred users. We actually converted to this centralized solution for the application about six months ago and the business owner has been converting locations two or three per week since.

I came into work to a my voice mail light... never a good sign. Apparently, they attempted to convert a large site the night before and had out grown the tablespace. "No biggie," I said and proceeded to increase the amount of allocated space. "We still can't get into the application and we cannot access any of the sites." This was very peculiar because this application has a different schema for each site and the only common information is stored in a "root" schema that manages some information on which databases are available. The user attempted to remove the affected schema from the application and I dropped it from the database, but this still did not allow the users into the application.

I'm not the most experienced person in the world, but I have dealt with quite a few applications during my career. I have never seen an application that just stopped working because of running out of space and especially not when more space was allocated. We contacted the vendor's support only to find out that their most experienced support analyst was out of town at a conference. The person we did get on the phone helped us check several things in the root schema, but couldn't seem to find anything that looked out of sync. The vendor suggested that we recover the database to prior to the load of the new schema. Knowing that the recovery wouldn't be a problem, I did have to warn them that since they had loaded the schema over twelve hours prior. This of course brings up other issues about notifications, etc, but that is a discussion for another blog posting.

Once we made the decision to go ahead and do the recovery, I looked at the dba_users view to find out when the affected schema had been created. Being the good DBA that I am, I copied the current data files to a backup location and started my RMAN recovery. Everything went perfectly and, as expected, the database came back up without the new schema. I called the application owner and told him to check it out. "It still doesn't work..." CRAP!!!!

This is the point when I start to wonder what is really going on here. It is also the point when the owner mentions that he actually loaded two locations the night before, not just the one. So, that would be recovery number two. I again figured out when the original schema was loaded and recovered to that earlier time. This time, when the owner tested the application, it worked. Okay, so at least we know which location caused the issue.

Users are allowed back into the application at this point and we plan for moving forward. The application owner says that he had tested that site in the test environment, but that he would do it again. When he tested it the second time, he found an error with the source data. Once the data was fixed, the load into test was successful. Somehow, I was talked into trying the load into production again since it worked in the test environment. I should have known better because as soon as he reloaded the site production was broken again. And that would be the explanation for recovery number three...

After the final recovery, we went ahead and copied the most current production data into the test environment. That would be refresh number one and when the data was loaded into the refreshed test environment, it broke it. At least we now had somewhere to try to figure out the issues. One more refresh and we waited to hear from the support person that new the most about the application. This is the part of the whole story that frustrated me the most. Once we had the right support person available, he was able to look and see that there was a data issue with one of the tables in the root schema. Deleted one row after loading the bad data and everything was fine. They deleted the row from the source data, reloaded into test, and everything worked. Being gun shy, we waited until the weekend to load the data into production just in case, but everything worked as expected, and he was able to load both sites without issue.

I learned a couple lessons throughout this ordeal. First, I had a gut feel that there was a data issue and not the tablespace issue that caused the application to not function. Unfortunately, I couldn't find any way to prove that and therefore had to go with the recovery that was being suggested by the vendor. This also leads to making sure the vendor has people available when needed to support issues. Of course, I didn't have anything to do about this because the vendor was chosen by the business. The other "stick to your guns" moment I had was when they wanted to load the data into production again after we thought it had been fixed. I should have insisted on a test refresh at that time to make sure we wouldn't break production again.

On the other hand, there were several good things that came out of this. My recoveries worked pretty well and I proved that I could go back to a point in time if necessary. The even better part of this was that the original data load was before the backup so we had to go back two nights and roll forward. I used to be a scripted backup and recovery person that could take or leave RMAN. After all these recoveries, to multiple points in time, I am an RMAN fan.

Well, I hope everyone had a great Christmas season, or whatever holiday you celebrated. Have a safe, happy, and prosperous new year. More posts soon. I really hope to get to this more often than I have recently. Also, make sure to look for information on Collaborate 10! See you in Vegas!

My First Two Weeks as an #OracleACE

2009-11-16T10:05:00.000-08:00

Life has been a little interesting for me personally for the last couple of weeks and by interesting I mean not necessarily in a good way. Things are much better now, but let's just say that rushing down the hospital hall with your wife on her way to emergency surgery is something I do not want to have to experience again... EVER! She is doing well now and is on the mend, but it was a very scary experience.

Almost lost is all of that was my being named an Oracle ACE! I am very honored to be added to this incredible Oracle community. I have met and gotten to know many from the ACE community over the last couple of years and already feel very welcomed. Thank you to Michelle Malcher, Oracle ACE and fellow IOUG Board of Directors member, for nominating me for the ACE award. Thank you also to George Trujillo. George is a former IOUG Board member and Oracle ACE. He encouraged me to get started on the Enterprise Best Practices SIG when it was in its early stages. I was looking for a way to get involved and George helped make it happen for me.

Oracle has recognized the importance of Users Groups and has an entire group dedicated to supporting us. I have found that I have become a better technologist and have quicker answers to problems since joining and becoming involved with the IOUG. I am looking forward to many more years in the Oracle community and look forward to continuing to contribute as a member of the ACE community.

Interview Questions for Collaborate #c10

2009-10-27T18:56:00.000-07:00

After much deliberation and thought, I finally decided to go a completely different direction for Collaborate 10 and do something for Professional Development. In the past I have presented on Clusterware, Data Guard, Oracle Portal, and, most recently, the PeopleSoft Management Pack for Grid Control. So, now for something completely different, I thought it would be fun to do something on interviewing.

Over the years, I have probably been on more than my fair share of interviews. Fortunately, so far, all have been by choice. The thing that always strikes me is the different types of questions that are asked. Some really make you think about your own experience and can trigger some good discussion. Others almost make you laugh. Not that I think it is a good thing to laugh at an interviewer, but at the same time they really make you wonder if this is a place you want to work. I have gathered several of these and have put together an abstract of the best and worst questions to ask or be asked in an interview. Some are down right funny and some should be standard because they show how people think and what their real experiences are.

If you want to hear what I think are good or bad, you'll have to attend Collaborate and check it out. What I would like are other examples from people that might be good to put into the presentation. Shoot me an e-mail or leave a comment and let me know if you have some really good, or really bad, interview questions. It should be kind of fun!

Back to the Real World and RAC Architecture

2009-10-23T12:35:00.000-07:00

I find it amazing that my last post was nine days ago already. It really seems like yesterday that I was still in San Francisco for OpenWorld. To close out that thought, the Aerosmith concert was pretty good, but not great. The could have played a little longer and done some more of the older stuff... (I was really hoping to hear "Big 10 Inch"). It is pretty amazing that they are able to put on that kind of a show at that age. I can only hope to be that active.

Thursday was actually not too bad mostly because I knew I had to be up early so I took it easier at the party than in past years. I was invited to the Inner Circle event for Oracle Fusion Applications because of my involvement on the Board of Directors with IOUG. This was my first event like this and it was very interesting. Most of what we saw and heard was not new to many in the audience because they had been taking part in the testing, but to me it was mostly new. The development and testing cycle they have been using was the most interesting part to me. It seems that Oracle is taking customer input very seriously. They have had users looking at the applications all along the way and taking the feedback and putting it into the applications. Hopefully that makes a better product for all of us.

For my company, the best part I heard was the ability to run Fusion apps along side existing applications. We are looking at adding some new modules to PeopleSoft and also bringing in some project management software. From what they showed us, the Fusion applications are built so we could bring in HR and Primavera on Fusion and seamlessly integrate with our existing PeopleSoft Financials system until we are ready to move that over. This sounds pretty nice and makes the switch seem easier than doing a full "fork lift" migration. I closed the trip with a PeopleSoft on RAC presentation and a late night trip home. All in all OpenWorld 09 was a great experience. Reconnecting with many people and putting faces to the phone voices make the trip worthwhile. Adding the content and the new people to be met are icing on the cake!

So, back to the real world. Part of the reason for attending the PeopleSoft on RAC session is that we are currently looking at replacing/upgrading our database infrastructure for PeopleSoft. I have spent part of the last couple of days researching and pricing a few different options for RAC. We currently have a shared HP Rx6600 that has eight cores and is virtualized. The database only has access to four cores and we are almost at capacity. The original plan was to purchase another Rx6600 and continue on that platform, but we are unsure of our growth over the next few years and think RAC would be a better option to give us the ability to scale as needed. That and the fact that the Rx6600 is Itanium are leading us to look at Intel based servers running Linux.

My main research points now are whether to go with more sockets with fewer cores or fewer sockets with more cores. I can see the pros and cons to both and will probably end up with fewer sockets with more cores, again because it gives us more flexibility for growth in the future. The other question is the interconnect. I'm sure gigabit ethernet would be okay, but I want to make sure there are no issues moving to a brand new platform. I'm including Infiniband switches into my design and figure I can always look at something less if we need to cut costs.

I have had to architect new solutions in the past and I always feel a little like I am putting my career on the line. That is where good research and good professional networks come into play. As long as I can back up my recommendations with multiple sources I am comfortable that things will work out. The user groups help here quite a bit (you didn't actually think I'd finish this without a user group plug). I have contacted three different people over the last couple of days for advice on my architecture. All have given me good advice and other sources of relevant information. I still haven't submitted an abstract for Collaborate 10 yet and they are due by midnight. Maybe this architecture research would be a good option...

Wednesday at OpenWorld

2009-10-14T16:26:00.000-07:00

So, today has been a busy day for me. Several meetings with Oracle execs. IOUG is very fortunate to be supported by such great people within Oracle. The day started with a breakfast meeting with a Director in the BI area. He will be supporting us in our regional BI efforts. Mark is totally supportive of what we are doing and will be helping to promote the event within Oracle.

My next meeting was with a senior manager in Oracle North American User Groups. I have worked with Tina over the last six months on our regional user group programs. She is totally supportive of us and we had a great meeting talking about where we are going with the programs and how to better support the members in the regional areas. This support structure within Oracle has been great for IOUG as we grow our membership. They give us the support we need while letting us remain independent. Excellent partnership...

IOUG president Ian Abramson and I met with a database development director to talk about offerings in the DBA developer space. Oracle has been working on some specific training for DBA's to introduce them to some of the development tools. This training is also good for developers, but in general they are targeting the training to the database development tools. APEX, PL/SQL, JDBC, C, C++, etc are all on the radar screen. Oracle feels they are losing some database development because people are setting up smaller database environments to do the development and then going to production with those instead of moving to the enterprise RDBMS. This makes complete sense and the IOUG can work very will with this team to promote the training and hopefully even deliver some of it.

I did make it to one session today which was PeopleSoft BI Tools. This was pretty good and was timely for me since we are looking at some PeopleSoft BI options at my "real job". There are some great tools out there that can be implemented quickly without building a bunch of new stuff. I also went to my first "unconference" session ever. This was on Oracle Warehouse Builder and even though a lot of it was over my head it was great information.

Larry seems to be finally winding down... wait, maybe not. Anyway, tonight is the big party. I'm looking forward to some more networking opportunities. Last night I was at the Oracle Publisher's reception and met someone from Land's End in Dodgeville, WI. He is an Oracle Press author and wants to get involved with the Wisconsin regional Oracle users group. Amazing how small the world really is and part of why I like being here. Limitless opportunities to meet with peers.

I should be closing out tomorrow as I head home to the family. A couple days away are nice, but it is always better to go home to Mary, Grace, and Abigail. See you tomorrow Sheetz girls!

OpenWorld Monday...

2009-10-12T08:42:00.000-07:00

Day two for me starts in the keynote with Safra Catz and Charles Phillips. We just had the intro from Judy Sims, the Chief Marketing Officer, and not it feels a little like a rock concert! Always interesting to see the new announcements and directions. Charles has been supporting the user groups a lot and has been at Collaborate for the last several years. It still amazes me the number of people they cram into this room. Great energy for a Monday morning.

Today is presentation day for me along with meetings with a couple regional user groups and special interest groups. These groups are the back bone to the user community and I encourage anyone that is interested in Oracle technology to find your local group and get involved. I'm looking forward to meeting these leaders and finding out how the IOUG can support and work together. My presentation this afternoon will be fun assuming the internet connection in the room works. I have screen shots for my demo, just in case, but that is far less interesting.

Safra Catz is talking about the amount of money they spend on R&D to help us get the right software that works. Interesting how they spend all this money to work on R&D and testing, and we still have lots to do when it comes to us. Charles just gave a shout out to user groups! They actually do listen and work with us so get involved and you can make a difference in the direction of Oracle. More later... demos coming.

OpenWorld Day 1 - User Group Sunday

2009-10-11T17:22:00.001-07:00

What a great start to OpenWorld. The User Group Forum has been a huge success. Great content and discussion around many Oracle technologies. My day started meeting new people and talking about the IOUG at the User Group Pavilion. If you need more information about any of the user groups, stop by the pavilion and check out the offerings. Linux and virtualization are two hot topics and the expert round table session proved that. We had two people from the Oracle Linux and VM teams that answered many questions about the environments. We also had several users on the panel that have experience implementing Linux in both virtualized and non-virutalized environments. There are several more sessions this week so check them out!

My other session this afternoon was a Town Hall meeting with the Oracle Software Security Assurance team. What a group of security minded people these guys were! Critical Patch Updates, My Oracle Support, and PSU's were all hot topics. Watch for announcements from IOUG for some future follow-up webcasts. We will probably be going more in depth into My Oracle Support and how to read the risk matrices for CPUs.

I'm off to listen to Larry Ellison and Scott McNealy. Should be in interesting presentation. More to come tomorrow. I have my PeopleSoft/Grid Control plug-in presentation tomorrow. I'm looking forward to a big crowd and some more good discussion. If you are around Moscone West tomorrow, stop by the User Group Pavilion and say Hi!

Oracle Critical Patch Update Questions? #ioug #oow09

2009-10-05T09:50:00.000-07:00

Ever since Oracle started a quarterly patch cycle for security patches, it seems that people are more confused than ever. Should I be installing these things right away? What about testing? Are there functionality patches that are "snuck" into the security patches? My company has a four month testing cycle and it would be a full time job just to keep up with quarterly patches so can I just do them once a year?

These are all questions that can be asked and answered at the IOUG/Oracle Software Security Assurance Town Hall. We will have Senior Managers and Directors from all the security teams to answer these and any other security assurance questions. This does not mean that they will be trying to sell you new tools, but rather will be there to give you as much information as they can about security and how Oracle is trying to make all the software as secure as possible.

Have you wondered where the Configuration Manager information that is gathered goes? Have you not allowed the installer to gather information or include your e-mail because you don't want a more spam? Make sure to come to the session and ask how this information is really used.

I will be moderating this session on Sunday, October 11 at 10:30 in room 2003 in Moscone West. This will be an excellent opportunity to discuss any Oracle security questions with the people that know, or can find, the information. The database, applications, middleware, and My Oracle Support teams will all be represented. Check out the session information here. Make sure to reserve your seat through Content Builder today and get your questions answered!

Clusterware Uninstall

2009-09-30T06:27:00.001-07:00

Sometimes I really miss the old days. Of course, someone my age saying that really doesn't mean much compared to someone that lived through World War II. When I started in IT there was no www, texting, cell phones in your pocket, or USB coffee warmers. My current lament is remembering the Oracle installer when it was all command line, java was another word for coffee, and an uninstall on Unix meant "rm -rf".

Where is this going? As the title of this post suggests, I had to recently uninstall Oracle Clusterware. I am working on a project to bring a new application on board and we are using an active/passive failover model using Oracle Clusterware. This model is something I have worked on several times and did a presentation at Collaborate 08 on the topic. So, I was going along installing away, having installed OCFS2, created voting and OCR disks and then installing Clusterware. The initial install went great until I realized that I had installed into a non-standard location compared to our other clusters. After the initial "Oh Crap" moment, I just figured I would uninstall what I did and install into a new home. This is where the fun began.

Of course, I realized this was not going to be as easy as ./runInstaller and remove the Clusterware install. I did not figure that it would cost me an entire day! After running the installer and removing the Clusterware home, I received an error saying that the directory could not be removed from the other node. No problem, I'll go remove it. So, I thought I was clean. No more Clusterware directory, no more Clusterware home in the inventory because I ran the installer. Boy was I wrong.

To make this long story a little shorter, I'll get to the end of what turned out to be four install/uninstall cycles. And, by the way, looking at Metalink, the only document I could find was removing from Windows. I am on Linux and it is not supposed to be this difficult! Okay, enough whining...

1. Make sure the crs and css processes have been stopped. This should have been as easy as going to /etc/init.d and issuing the init.crsd stop and init.cssd stop, but this did not stop all processes. After all the deletions and cleanup, the only way I could really get rid of the processes was to kill them as root.
2. After issuing the stop commands, delete the init.crs, init.crsd, init.cssd, and init.evmd files from the /etc/init.d directory on both nodes.
3. Edit the /etc/inittab file and remove the lines at the bottom that start the init.evmd, init.cssd, and init.crsd
4. Remove the init.crs symbolic link from /etc/rc.d/rc0.d, rc1.d, rc2.d, rc3.d, rc4.d, rc5.d, and rc6.d. This link will start with K## where the ## is a number. Mine looks like K96init.crs and is a symbolic link back to /etc/init.d/init.crs
5. Remove the physical Oracle home directory that Clusterware was installed in.
6. Reboot! I know, it seems very Windowsish, but it was the only way to make sure all the pieces of crs and css were out of memory.

Having done all of that, I was finally able to reinstall into my new Oracle home. So, the moral of my story... make sure you are following your own standards before starting!

See you in 10 days at Oracle OpenWorld. Don't forget about User Group Forum Sunday.

Linux and Virtualization at OpenWorld #oow09

2009-09-23T05:01:00.000-07:00

I had a very productive discussion with several people from the Oracle Linux and Virtualization team yesterday in preparation for the Linux Roundtable session on Sunday at OpenWorld. The session is going to start with an introduction to the Linux team and the Virtualization team by the directors of those technical organizations. We will then have questions and answers for them along with the remaining panel members who are all presenting topics on Linux. This should be a lively discussion and I'm looking forward to facilitating and getting some of my own questions answered.

Topics for the session will include validated configurations, why the Virtualization team chose Xen, patching cycles, how are bugs reported and fixed, and anything else the audience chooses to throw at the panel. This really should be a good combination of how Oracle thinks the technologies are being used and what they are doing to facilitate that use complimented with how real world users are doing. Don't miss the session for anything Oracle on Linux and Virtualization. The session is Sunday, October 11 at 10:30.

For the remaining Linux and Virtualization topics check out this post which lists all of Oracle's sessions. If you are coming to OpenWorld drop me a comment and let me know. I'd love to touch base with people. You will also be able to find me pretty much all day at the User Group forum on Sunday.

For now, I'm on the train on my way to Chicago for our inaugural IOUG/Chicago Oracle Users Group Regional BI Event. Should be very interesting and I'll let everyone know how it goes. Check out the upcoming events here. Upcoming sessions include Pittsburgh and Boston and plans are in the works for several others.

OpenWorld 09: T-minus one month!

2009-09-10T08:10:00.000-07:00

Only a month remains for me to finish my presentation... yikes! Actually, the biggest problem I am having now is an open Service Request that is in development for a PeopleSoft Plug-in issue after we upgraded our PeopleTools version. Hopefully that will be resolved in the next month. It will be pretty hard to present, "look how cool this is" if it isn't working properly.

Sunday, I will be part of two sessions that I am pretty excited about. Sunday is User Group Forum day at OpenWorld and I promise there is something here for everyone. I am facilitating a Linux Roundtable discussion and a Software Security Assurance Team Town Hall meeting. Both should be fascinating. The Linux discussion will be anything to do with running Oracle technology on Linux. We will have someone from the development teams at Oracle for Linux and Virtualization, along with a some other customers that are Linux users. The Software Security Assurance Town Hall is your chance to ask all those questions you have not had a chance to ask about Security Assurance. We will have Oracle managers and directors that have responsibility for security development. Oracle Configuration Manager, My Oracle Support, Critical Patch Updates, and application security are all on the table for discussion.

Those are just my sessions (not that I'm biased of course) and all the users groups have sessions going on. Make sure to sign up for the OpenWorld Content Builder and reserve your seat for those Sunday sessions. Some of them are filling up already! There are sessions on upgrades, Exadata, BI, Spatial technologies, Content Management, RAC, APEX, Portal technologies and more. I'm starting to sound like a TV pitch man... "and that's not all!"

Of course technical content is only half the story. Who isn't looking forward to Aerosmith and being able to network and meet new people. I know I'm looking forward to catching up with some people I haven't seen since Collaborate! One way to do this for Oracle bloggers is at the Blogger Meetup that Alex Gorbachev has picked up the organizer responsibility for (thanks Alex!). Should be a good opportunity to put a face with a blog.

I better go see if I have any updates from Oracle Support... hint, hint! See you all in San Francisco.

Why can't I add a unique index?

2009-08-25T07:41:00.000-07:00

Adding unique indexes once data is already loaded into a table can prove to be a bit of a headache. The question of how to clean up duplicate keys is one that I have had posed to me many times and it seems somewhat simple. I guess anything is simple if you already know how to do it, so to that end, I thought I would write my solution.

This became an issue again recently because the UNIX admin where I work was creating a database to store system statistics. He has daily flat files that he wants to load and knew the basics of creating a database. As he was loading the data it occurred to him that he did not want to be able to load the same file twice and he decided to try to add a unique index. Funny thing was that what he was trying to prevent had already happened and he received an error trying to create the unique index. His next stop was my desk to ask why the error was happening and how to fix it.

The first thing we needed to do was to figure out which rows were duplicated. The following SQL gave us the date that was loaded twice:

select collection_date, server, count(*)
from system_stats
group by collection_date, server
having count(*) > 1;

This gave us a list of machines for one date with 2 rows each. Now we knew what date was loaded twice, but since they were exactly the same, how do we get rid of one row for each key? This took creating a temporary table to hold the records we didn't want to delete.

create table hold_records as
select distinct * from system_stats
where collection_date = ;

Checking the hold_records table shows us that we have exactly half the number of records for that date and each server only has one record for that date. Now we clean up the data for that date from the main table.

delete system_stats where collection_date = ;

Now we put the data back:

insert into system_stats (select * from hold_records);

Voila! One record per server for that date and the unique index could be created. Again, this is not really rocket science, but I am surprised at how many times I have been asked to do something similar. Hopefully it will save someone else some time in the future.

More Manageable or Different Management?

2009-08-18T07:05:00.000-07:00

With every new release of the Oracle Database, we are told that there are new manageability features to make our lives easier. Larry Ellison even told managers at one point that they wouldn't need a DBA anymore. I have been working on implementing the PeopleSoft Plug-in for Enterprise Manager and wrote a paper about it for Collaborate. I am presenting the paper again at OpenWorld and have been updating it with new information. This is where the question that is the title of this entry comes in. Do these tools make things more manageable or just force us to focus on a different kind of management?

Implementing management tools is only half the story. If monitoring was as easy as run setup and you're done, I wouldn't have needed to write a paper or do two versions of the presentation. What to monitor, what to alarm on, what level to alarm at, and who needs to be notified based on what the alarm is... All this functionality is built into the tools, but figuring out exactly how to set it all up is the trick. On some level I feel like writing shell scripts might be easier. Tools become a better bet when there are changes to the environment. Having one location to go to add new environments or take care of changes, does start to make tools worth while.

As with most application implementations, that is only half the story. Security has also been a challenge to set up because of the normal implementation of Enterprise Manager. Extending one tool, that is usually used for one purpose, in this case the database monitoring, to include other purposes, such as application monitoring, brings to light many short comings in the original implementation. In our case, our DBA's were Super Adminstrators with access to all targets in Enterprise Manager. Of course, from an application adminstration perspective, this causes separation of duties issues. So back to the security drawing board. Once this is designed properly, it is not that difficult to maintain, but building the proper design takes time.

Bottom line for me is that I still believe that monitoring tools and new manageability features make our lives easier in the long run. The moral of the story (and my presentation) is to plan up front and make sure you have a good design in place. I'm still working on my implementation because I didn't know what I didn't know when I started. Hopefully my paper and presentation can save someone some time. Even more interesting will be the new manageability features introduced in Oracle Database 11g Release 2. Hopefully more on those to come...

IOUG at Oracle OpenWorld

2009-08-09T11:23:00.000-07:00

So we are about nine weeks out from Oracle OpenWorld 2009 and planning is well under way for the Users Groups. Several years ago, the Sunday before OpenWorld opens became Users Group Sunday. This year there is more participation than ever and an even higher level of technical content. The Independent Oracle Users Group has seven dedicated rooms and two shared rooms with the Oracle Development Tools User Group. Topics include 11g upgrade, Business Intelligence, Exadata, RAC, Content Management, Spatial Technologies, SAP and much more. Most of the Oracle Users Groups are represented on Sunday so make sure you include the Sunday sessions as you are making your travel reservations for OpenWorld.

Besides the Sunday program, the Users Groups also had an opportunity to choose some of the non-Oracle employee technical content for the regular conference sessions. Hinting, compression, Critical Patch Updates, 11g new features, 11g upgrades, Warehouse Builder and many more sessions were actually chosen by IOUG. Seasoned presenters, authors and Oracle Aces are all included in the line up.

And to top it all off, if you would like to get all of this for free, here is an option if you are lucky enough! Review the new Oracle Support here and be entered to win a free pass.

More details about some sessions of interest as we get closer. Hope to see you all in San Francisco in October!

Funny T-shirt or Life Lesson?

2009-08-03T21:37:00.000-07:00

I think most men have that one t-shirt in their dresser that their wife has said, "that t-shirt is falling apart. Why don't you get rid of it?" I think I have three or four, but one of my favorites is an old Embarcadero Technologies shirt that has a picture of a woman dressed in black crying over a casket with someone tapping her on the shoulder. The caption says something to the effect of, "I know this is a bad time, but did he happen to tell you his database admin password?" I always thought it was kind of a funny shirt given how that is always the worst case scenario that we as database administrators can think of. I used to have a manager that would tell us to document our processes and procedure just in case someone got hit by the "chicken truck".

Unfortunately, I experienced this in real life last week. One of my coworkers passed away unexpectedly last weekend. I had just begun to work with him recently and so I really didn't know him that well. He was actually at an Oracle class when he got sick and had to come home. I had been working with him to help him better understand the Oracle database. In fact, I was supposed to be helping him on a new project when he came back to the office. Not surprisingly, there were some tearful eyes and heavy hearts when we all got the news on Monday morning.

Once I got over the initial shock of wondering how someone roughly my same age would go from relatively healthy to passing away, the inevitable question became, "what about supporting the organization during this sad time?" Our organization is split up so the databases that my coworker was supporting, I had never seen before. For the most part, I was able to figure out what was going on and make sure there was enough disk space, backups were running, and most things seemed in order. What we did figure out though is how much documented standards, processes, and procedures would have made the transition easier. One thing I will be discussing with management is consolidation of tasks and a set of standards that can be documented and shared with the team.

The moral of the story is pretty much what most of us already know. Document everything and make sure that the documentation and passwords are in a safe, secure place that is accessible to those that need them.

God speed Kurt... you will be missed.

Voting DOES Make a Difference... In the Oracle Community

2009-07-01T07:14:00.000-07:00

Around election time, it is not uncommon to hear someone say, "I'm not going to vote because it never makes a difference." In these uncertain times, my response would be, "when is your vote more important?" Leaving politics aside, there are a couple places that your vote makes a difference right now!

User Group Survey
Several user groups, including the Independent Oracle Users Group have teamed with Oracle to find out how important your user group experience is to you and your company. The purpose of the survey is to show Oracle that their support of user groups is important and to allow the user groups to be able to better serve you, the member. Take a few minutes and let the community know your thoughts on user groups. You have until Friday, July 3 to make your voice heard. Take the survey here:

http://www.oraclesurveys.com/se.ashx?s=251137454671BECB

Oracle Mix Voting
Oracle OpenWorld planning is in full swing for the show in October. Did you know that you can have an influence on the content that is provided? Create an account on Oracle Mix if you don't already have one and vote for sessions that you would like to see. The link to vote is below. I voted for six sessions so far, but haven't looked at all of them. Check it out and make your voice heard so the conference is the best experience for you.

https://mix.oracle.com/oow/proposals

OpenWorld Sunday
Speaking of OpenWorld, while you are registering and making your travel arrangements, don't forget about the User Group Forum on Sunday. Several of the User Groups are presenting some of their best technical content. IOUG alone is participating in nine rooms with five sessions in each. Learn about everything from RAC, Linux, BI, and Exadata to Content Mangement, BEA/Portal technologies, Spatial technologies, and Application Express. Check out all the session on Oracle's Content Catalog at the link below. Search for Session Type of User Group Forum and SIG Meeting to find the content on Sunday. Make sure to be at OpenWorld early and take part in some great technical content.

http://www35.cplan.com/cc221_new/newCatalog.jsp

Monitoring the Monitor

2009-05-20T06:51:00.000-07:00

Occasionally we hear about government officials that are supposed to be watching out for our good and instead look out for their own. When this happens, people will sometimes ask, "Who was watching the watchers." I was reminded of this last week when our Oracle Enterprise Manager Grid Control server lost network connectivity and we were not notified. A very large thunderstorm passed through and we also found out the next morning that we had switched to generator power at least three separate times during the night. Sometime during one of those switch overs, the network interface lost contact.

Of course we did not realize it was down for about 14 hours. The company uses HP's Network Node Manager which raises alerts to the on-call help desk person, but for some reason, no call was made to the system administrator that the server was unavailable. Not only do we use Grid Control to monitor all of our Oracle and People Soft systems, but we also use it as a centralized job scheduler. Once the server was available again, I figured out that we missed about 150 jobs. No backups, no disaster recovery log maintenance, no nightly application maintenance jobs, no nothing for fourteen hours. The better part of the next day was spent running backups and getting our Data Guard instances caught up since the archivelog destinations filled up.

So now what? We considered writing a shell script and using cron on one of the production servers to ping the Grid Control server and e-mail if it was not available. Obviously, this is a bit "patchwork" but should at least tell us if the server is unavailable. Another option would be to set up Network Node Manager to page the system administrator directly if a server goes down. This would be my choice, and I plan to work to get this implemented, but at this time, I don't have access to that system. In the meantime we decided to use our test Grid Control system (yes, we have one) to monitor the production Grid Control server. Of course this means a separate Agent installed on the production server, but at least we will have visability into the system should something happen agin.

The question through all of this is how much is enough? I guess you could ask the same question about watching our government representatives... although I think they need to be watched even closer!

Collaborate 09 - Wednesday

2009-05-06T07:11:00.001-07:00

I wasn't able to write an update yesterday, but not because there wasn't anything to write about. On the contrary, I was so busy that I didn't have an opportunity to sit down long enough to write. The week has been very productive so far and will continue to be. Of course, everyone is looking forward to the party tonight and Universal Studios Islands of Adventure.

A few things that I didn't cover last time. Sunday evening we had the IOUG SIG Welcome Reception. This was very well attended and was a great opportunity to meet some new people and reconnect with people I haven't seen in a year. The Special Interest Groups each had a table and many people were interested in what the SIGs are doing. I was talking to many people about the Linux SIG and we should get some new blood into the group. Many thanks to SAP for sponsoring this event. We are always looking for vendors and friends that are willing to sponsor these events for us and SAP always seems to come through.

As one of the new IOUG Board of Directors members this year, I had my orientation yesterday to learn some more about my new responsibilities. I'm looking forward to the challenges and exciting times over the next two years of my term. My role on the Board will be Director of RUGs and SIGs. Joining these two groups under one Director makes a lot of sense. We should be able to jointly advertise for both groups as well as draw on each other's experiences and resources. Between the RUG Leader meeting yesterday and the SIG Council meeting this morning, I can already tell there is a lot of energy and great ideas from both communities.

The technical content this year is as good or better than it ever has been. All the new features of products and the new stuff people are doing with old things have been very interesting. Of course I have also been immersed in SIG meetings and Board of Directors meetings. I'm now in the last session of the day and learning about recovery scripts. An important topic, no doubt. It is very hard to believe that tomorrow is the last day already. I'm looking forward to a great time tonight with new and old friends. My presentation is finally tomorrow at 11:00. I'll be talking about the PeopleSoft Management Pack for Grid Control. I'm the very last session of the conference so I have a few giveaways to try to lure people in. Actually, I'm hoping that will get me some good reviews on my surveys!

See you on the roller coasters!

Collaborate 09 - Day 1: Charles Phillips

2009-05-04T08:20:00.000-07:00

What a great start to Collaborate already! Charles Phillips, President of Oracle, had our keynote presentation this morning. Some highlights from the talk include some new releases, extension of support and a demo of Beehive. Charles mentioned that PeopleSoft 9.1 should be available around OpenWorld time. Not much more on that because the big deal was the announcement of EBuiness Suite 12.1. Along with that comes 9 new products that have been updated to 12.1 and 14 production that can run on 11i10 with no upgrade required. One of the main threads that came through was Oracle's sensitivity to the economic times and that not everyone can afford to upgrade. Along those lines he also announced extended support for several products without any additional cost. These included 10gR2 database, which was extended to July 2011, PeopleSoft 8.9, which was extended to Jun 2011, and EBusiness 11.9. I neglected to get that date, but they basically added a year. There were a couple other products that were extended, so I guess it's time to check out the support grid again.

Much of the discussion covered integration and collaboration. The Application Integration Model (AIA) has been expanded to include more applications and uses Oracle Middleware and web services to integrate applications. Oracle is selling the Process Integration Packs (PIP) in order to automatically integrate the applications. These are used to let Oracle worry about your integrations. Sounds great... I'd like to see it in action.

Speaking of seeing in action, there was a demo of Beehive. For many of us, we have heard the term, but are not sure what Beehive is or does. Collaboration and projects are becoming a big focus and this helps to address those issues. The big announcement today for Beehive is the release of the Team Collaboration module that goes GA tomorrow. This allows project teams to create Wikis, team calendars, versioning and check out of files all through one server. Beehive alos includes chat and e-mail and can integrate with other e-mail solutions. The demo was pretty cool and Charles said that Oracle is running this with ten to fifteen thousand users being hosted on one server.

One interesting side comment was about Exadata. Charles was talking about Exadata and how it currently uses Linux. Then he said that the Sun acquisition will not have any effect on Exadata, but that there will probably be a Solaris version at some point... interesting, very interesting.

I am now in the Darwin Awards for IT security. Some of these examples are a little funny and even more scary. I just hope never to be a recipient! More about Collaborate later.

My (Almost Solid) Collaborate 09 Schedule

2009-04-22T10:52:00.000-07:00

With less than a week before Collaborate 09 in Orlando, I have been working on the schedule of sessions that I don't want to miss and trying to fit in as many as I can around meetings. Every year, I have the same issue. Do I attend sessions for things I'm working on now to see if I can better my current processes, or do I look for new features and future products? This is actually a good problem to have and is an indication of the level of content that will be available.

Meetings for Special Interest Groups and now the IOUG Board of Directors take about half of my time, but that leaves plenty of time for some great session content. I will be dipping into some of Quest' International's content this year also since I am back to supporting the PeopleSoft world. Besides PeopleSoft I am trying to balance my time between new topics and improvements to current projects I am working on.

I'm especially looking forward to the Darwin Awards for IT Security and New and Improved Features in 11g RMAN. There should be several things in both of these that I can use right away back at the office. Backup scripts without restore scripts??? Who would do that? I guess that will be one session I will have to check out and make sure my restore scripts are up to snuff. Of course, my own session, which happens to be the last technical session on Thursday will be of interest to anyone that supports PeopleSoft. We are implementing the PeopleSoft Management Pack for Grid Control and my presentation covers some of our headaches and successes.

Collaborate wouldn't be complete without catching up with all the people I have become friends with in the past and making some new ones. Talk about value for the money. My contacts from past conferences have saved me a ton of time and my company some cash in the process. If you think you might be missing out because you won't be in Orlando next week... you're right! You still have two days to get $200 off the on-site registration rate and discounted hotel rates. Make sure you don't find yourself stuck in a situation that you could have learned about by attending Collaborate 09!

And when Collaborate is over.... I'M GOING TO DISNEY WORLD!

I plan on Tweeting from the conference and will hopefully find some time to write blog entry or two. I hope you follow along! Find me on Twitter... toddsheetz.

"Virtual" Test

2009-04-22T09:17:00.000-07:00

Budgets, budgets, budgets. We all have to deal with budgets. Making sure we can pay the mortgage, make the car payment, and feed the family fit into the home budget. For anyone that is a member of an organization, such as IOUG, we may want to start a really cool project, but being a volunteer organization, the budget will not support it this year. Even sending my daughters to school has become interesting as the private school system that they attend has run into budgetary issues. Most everyone in the technology field, definitely has run into budget constraints at work. In my case, the most recent budget "issue" has been creating a test system for our latest project.

A project was started to consolidate some remote MS Access databases into a centralized Oracle database. This is a vendor application and either installation method is supported. As the application becomes more mission critical, however, the decision was made (and the correct one, in my opinion) to consolidate to our Network Operations Center. The issue that came up though was dollars. When the original budget was written for the project, dollars for a test system were forgotten. The production system was budgeted as a two-node cluster to support two different application databases. This was fine and should support what we want to accomplish, but test was completely overlooked.

So, how do you create a test cluster to support a production implementation when dollars are limited? In our case, the decision was made to purchase one physical server and user Oracle VM to create a test cluster. Not exactly the same as production, but at least we will have somewhere to test out operating system, database, clusterware, and ASM upgrades and patches before going directly into production.

Of course, we have had issues. There is a larger VM learning curve than we expected especially as it relates to storage. We are also feeling our way through resource allocation. Originally we underallocated CPU resources and couldn't install anything. The virtual guest machine showed that the CPU was pegged, but the physical host showed no activity. Allocating an extra virtual CPU took care of that issue, but we still had issues installing and configuring ASM. The normal installation of 11g and configuration of ASM was hanging and showing full CPU utilization, although it was creating the instances. My coworker has spent a couple days with Oracle support trying to figure things out and finally ended up creating everything manually instead. Just one instance of how virtualized environments are not quite as seamless as the are touted to be.

In any event, we have the viritual servers up and running. So now we need to create the databases and away we go! That is just one example of how, if necessary, budgets can be saved. Of course, we haven't proven that we can support any kind of test load on this architecture, but we should at least be able to see if the applications still function after an upgrade or patch.

One final plug, by the way, for the IOUG Forum at Collaborate 09. If you are looking for more ways to save budget dollars, there are lots of tips and tricks you can learn from your peers. I'm hoping to learn more about virtualization and how we can better tune our test system. Since budgets are tight everywhere, make sure you register online before April 30 to save $200. Also, the conference hotel rates have been discounted. There is not better bargain for training than learning from people that have actually used the technology...

Collaborate 09 - Early Bird Registration and Special Offer

2009-04-01T14:19:00.000-07:00

Okay, so I know I already posted today, but I don't think I put enough information about Early Bird Registration for Collaborate 09. You only have one more day to save $400 and get some extra stuff by using one of the special registration codes.

Register for the IOUG Forum by this Thursday, April 2 at http://w3.ioug.org/COLLAB09/LinkedIn using promotional code AD01 (choose this code during the demographic section – step 4 – of the registration process). You will receive a free Oracle Technology Best Practices Booklet and have your name entered into a raffle for an Apple iPod Touch. Of course, you also get access to the entire conference – more than 1,600 sessions! Oracle database, RAC, PeopleSoft, eBusiness Suite, Grid Control, and much more all in one place. All this training for one low price will save your company money by not having you go to multiple classes and you will be able to put the lessons learned into practice as soon as you get back to the office - and don't forget the networking.

Okay, enough marketing! I only blog about this because it is very important to me and I have benefitted from it for many years. I promise, you will not be disappointed!

Hope to see you in Orlando and make sure you get that registration in by tomorrow!

April Fools Thoughts...

2009-04-01T07:06:00.001-07:00

I am in a RAC Workshop at the Oracle office in Milwaukee today. We are still in the set up phase so I have a few minutes. This should be either a good follow-up to our recent Clusterware install, or a rude awakening that we need to start over! Let's hope for the former.

Yesterday and today we were in the process of installing a kernel upgrade to our cluster nodes. I have used Clusterware and OCFS on Linux in the past, but never with ASM. The trick was figuring out the order between installing the kernel upgrade and the ASM and OCFS drivers to match the kernel. To add to the trouble the QLogic driver needed to be reinstalled. Lots of learning experiences! The pleasant surprise to me was that once we got everything upgraded on the first node all the pieces, including Clusterware, started normally, even though node two was not yet upgraded. This is a nice upgrade from my last experiences with all this when we had to take node two down before OCFS and Clusterware would start on the upgraded node. Nice to know that future upgrades should allow for much shorter downtime during the failover.

A reminder that early-bird registration for Collaborate 09 closes tomorrow. Hopefully everyone can make it down to Orlando in May. As a member of the Conference Committee, we are hard at work reviewing papers and presentations. The technical content should be excellent as usual. I have been reviewing papers for the Achitecture track and have read some great papers. Since I finally finished my own, I have more time to read others. Now I just need to complete my presentation... details, details.

IOUG members are reminded that the Board of Directors elections are open until Saturday. This is a bit of a selfish plug since I am running for the Board. Most importantly, make sure you vote. A vote for me is very appreciated! Next year's Board promises to be a very good group of people and will continue move IOUG forward.

The workshop is starting... better go pay attention!

Clusterware and PERL

2009-03-28T16:32:00.000-07:00

A couple weeks ago, I wrote about my new project at work. We are implementing single instance databases protected by Oracle Clusterware. Eventually, the thought is that we will upgrade to Real Application Clusters, but for now we should be able to support the applications with single instance databases.

This week we completed the set up of the cluster after a few technical delays. Our final hurdle seemed to be starting the database using Clusterware. Part of the registration process for a resource in Clusterware is to tell Clusterware what script has the actions available for that resource. Oracle provides action scripts for resources such as listeners, databases, and Enterprise Manager Agents in a couple white papers that need to be copied out and implemented. The action scripts that are provided in the papers are in the form of PERL scripts so they are portable. PERL is one of those things that I have always meant to learn, but never really made the time to pick it up. The database action script was failing when we tried to start the database, but we were able to start the database on both nodes without using Clusterware.

With some help we were able to add some logging to the script and sent some of the variable values to a text file. Using this method we were able to narrow down the issue to the check subroutine in the script. Further analysis narrowed the problem down to the following line of code:

$check_proc = qx(ps -aef | grep ora_pmon_$ORACLE_SID | grep -v grep | awk '{print $8}');

When I cut the line out of the script and ran it from the shell, it returned exactly what was expected. When we logged the output from in the PERL script, it returned the entire line from the process. For some reason, the awk part of the command was not being executed. Finally, I realized that PERL was treating the $8 part of the awk command as a variable, but since there was no variable for 8, the awk was not returning the eighth field as expected. A simple backslash in front of the $ took care of the issue:

$check_proc = qx(ps -aef | grep ora_pmon_$ORACLE_SID | grep -v grep | awk '{print \$8}');

About the same time that I figured this out, the person I had been working with at Oracle informed me that I had been using an older version of the script. They have since replaced the line with a better option:

$check_proc = qx(ps -ae -o cmd | grep -x ora_pmon_${ORACLE_SID});

Obviously this is much cleaner and accomplishes the same thing. So I learned two things from this experience. Number one, always check the dollar signs in a PERL script and number two, make sure you have the latest version of the script!

Why Collaborate?

2009-03-24T10:46:00.000-07:00

I have mentioned in previous posts that I will be presenting a paper at Collaborate 09 in May. This will be my third year presenting and I have found it very rewarding. I attended IOUG events back when the conference was called IOUG Live, and it has only gotten better since I have been involved with presenting. So why should someone go to Collaborate instead of taking an Oracle class? Why should you fight to go when training budgets are being cut? I have several reasons.

Content
Collaborate is one of the few places that you can find the best technologists in the Oracle community all in one place. These are people who are really doing the work and managing the projects to implement and support Oracle technology. Go through your list of Oracle authors and chances are they will be there. Tom Kyte, Rich Niemiec, Michael Corey, Ian Abramson, Michael Abbey, Carey Millsap, Jonathan Lewis... and the list goes on. There will be 350 sessions and that is just from the IOUG Forum. Add in OAUG and Quest International and the options are almost endless.

Are you having issues with Real Application Clusters? Can't quite figure out how to get PeopleSoft monitored with Enterprise Manager? Need help designing a process to patch eBusiness Suite? Starting your first BI project and don't know what BI means or what it has to do with stars? Answers to these questions and more can found at Collaborate. Many times I have been sitting in a session when the speaker said something and I thought, "so that's how you do that!"

Networking
I already listed some of the well known authors/presenters that will be in Orlando this year, but that's really only half the story. Ask the Experts sessions, topic specific orientations, the IOUG/SIG Welcome reception, "Tuning" Open Mic Night, lunches and receptions are all great places to meet new people that are doing some of the same things you are doing. Personally, there have been many times that I have been stuck on an issue, or just wanted to bounce an idea off of someone, that I have e-mailed or called someone I have met at Collaborate or through the IOUG. The network of people that I have connected with has been a huge time saver for me and a money saver for the companies I have worked for.

Cost
Recently I did a comparison between what it would cost for me to go to and Oracle instructor led training course instead of going to Collaborate. To just attend the conference, you could get away with traveling Sunday to Thursday and only pay for four hotel nights at one of the Conference hotels.

Collaborate Early Bird Registration:	$1,335
Airfare:	$200
Airport transfer:	$29
Hotel - Four nights@$140 + tax:	$625
Food:	$100
Total:	$2,289

Oracle Instructor Led Class:	$3,000
Train Fare:	$44
Cab Fare Roundtrip:	$40
Hotel - Four nights@$160 + tax:	$739
Food:	$125
Total:	$3,948

As you can see, you can save at least $1700 by going to Collaborate instead of an Oracle class. And that is if you have access to taking the train to an Oracle education site on Monday morning. If you have to fly, there would be added cost for the flight, plus an extra hotel night. Collaborate is four days and the Oracle class is five days, so that is part of the difference.

If you add a Sunday University Seminar for $459 and one extra hotel room night at $155, you would still only be spending $2903, a full $1000 less. You could also add a Thursday University Seminar for just the cost of the session and not add to the travel budget. The University Seminars are targeted deeper dive training. I would highly recommend trying to go to at least one of them.

Why Collaborate?
You can now see what I Collaborate! You could say it has become a little bit of a passion, but it is something I very much believe in. Check out the registration link at www.ioug.org/collaborate09 and look for the top ten reasons to attend. Also, you can find a proposal to give to your management on why it is important to be at Collaborate. Hope to see you in Orlando!

Until next time... stay clean!

New Clusterware Project

2009-03-13T20:23:00.000-07:00

For the last week, I have been working on our newest project. We are consolidating an application that is currently in pilot mode across several locations. Some of the current databases are SQL Server and some are Oracle. We were brought into the project a little late, but have recovered nicely to work out an architecture that we think will be a good starting point for growing this environment.

I have learned a lot about project teams and technology through this process. I work for Veolia Environmental Services and if you have heard of our company, you know it is huge. We are starting to do Information Systems support for some of our sister companies, and that alone is proving to be a challenge. Trying to meld different corporate cultures is bad enough and then add in the technology layer and it can be really fun! I may cover the culture clashes some other time, but for nowI thought it might be interesting to go over the technology.

We started planning on using Oracle Standard Edition database and Real Application Clusters running on Oracle Enterprise Linux using HP hardware with Intel x86-64 processors. Many of our discussions centered around the version of Oracle and whether or not to implement RAC. We made the decision to stick with Standard Edition even though all of our current implementations are Enterprise Edition because of budgetary concerns. The plan is to move to Enterprise Edition in the future so we can standardize our licensing, but for now, we have a limited budget and are implementing Standard Edition. We also decided to forgo RAC and user Clusterware to protect the databases using active/passive clustering instead of active/active. Our timeline was very short and I have used Clusterware in this fashion before so this helped to shorten the timeline, but we also don't really know how much traffic we are talking about so RAC may not even be necessary.

So we have Clusterware installed and running, Automatic Storage Management is running and storage is allocated, the RDBMS is installed and patched, and we are ready to create the databases. Next comes the fun part... pushing power buttons and pulling power cords to make sure it all works as planned! This has been an incredible learning experience and should continue to be. Next is building our test environment using Oracle VM! Let me know any comments if you have had similar projects or want to chat about what we're doing.

Until next time... keep learning! Oh, by the way, I'm sure there will be lots of good sessions on Clusterware, RAC, Linux, etc at Collaborate 09 this year.

Oracle Magazine Interview

2009-03-12T14:09:00.000-07:00

Today was kind of an exciting day for me. I was interviewed via webcam and Skype for Oracle Magazine. They do a Linux issue every year and they have started doing a feature on the user community. I am the current president of the Linux Special Interest Group (SIG) for the Independent Oracle User Group and they asked if they could interview me about the SIG and our relationship with Oracle. I was contacted by Jeff Erickson at Oracle Magazine through Rich Schwerin, who I mentioned in an earlier blog post. Rich is the Senior Product Marketing Manager in the Linux and VM group at Oracle. We talked for about a half hour about what the SIG has done in the past, our future plans, and how Oracle is doing in the Linux and support areas. I won’t ruin it for you and you’ll have to check back to find the link to the video, but I really appreciate the opportunity to talk about the IOUG and the Linux SIG.

For part of the interview we discussed personal and professional opportunities that come from involvement, not just membership, but involvement, in the user group community. Personally, I have made some great friendships and have people that I talk to, either on the phone or electronically, regularly. Professionally I have made some networking relationships that have saved me several times. This is where involvement really comes in. I started with IOUG as a member attending the conferences. That was great, and I learned a lot, but it was always lacking a little something for me. Once I started actually getting involved and taking a leadership role in the SIGs, the experience changed. The interview also covered my experience with Linux and the relationship with Oracle. At my previous company, we implemented Oracle Clusterware on Red Hat Linux. Those networking relationships really came in handy there! Now, my coworkers and I are embarking on a Clusterware project using Oracle Enterprise Linux. I'm looking forward to getting back into it as well as having something to share with the Linux SIG.

So, I will post the link to the video in a couple months. Hopefully I didn’t look too geeky. In the meantime, make sure you are getting the most out of your career by getting involved. And don't forget that Collaborate 09 is coming up in May. There will be several sessions on Linux and over 350 sessions total just from the IOUG Forum. IOUG is always looking for good volunteers that are willing to step up and take a leadership role. The SIGs are a very good way to do that. Who knows… maybe next time you could be the one being interviewed for the magazine! Speaking of that, make sure you check out IOUG President Ian Abramson’s blog today. He is also talking about being interviewed by the media. He also has a good post about some of the inner workings of the IOUG Board from yesterday.

Collaborate 09 Paper: PeopleSoft Management Plug-in for Oracle Enterprise Manager (aka Best Laid Plans)

2009-03-09T18:44:00.000-07:00

“The best laid plans of mice and men often go awry.”

I’m working on writing my paper for Collaborate 09, and for some reason, this phrase keeps coming back to me. I actually looked up this quote and it is often misquoted, including in my opening sentence. The quote is actually from a poem by Robert Burns, called “To a Mouse” and the quote is actually, “The best laid schemes o’ mice an’ men, Gang aft a-gley.” Burns, incidentally, also wrote Auld Lang Syne. My best laid plans included actually having my white paper and presentation completed for Collaborate 09 before they were due. This year my presentation is “Implementing the PeopleSoft Enterprise Environment Management Plug-in for Oracle Enterprise Manager.” I did get an early start, but have been working on this for about six months and have been unable to complete the project.

Issues started almost immediately with the installation process. We are an HP-UX shop running on Itanium CPUs. The installation of the plug-in to the Oracle Enterprise Manager Management Service failed with a password issue. After several weeks, Oracle determined that the installer was having issues on Itanium. They created a patch for the installer that allowed me to install the plug-in and I was finally on my way. You will have to wait for my paper and presentation for the details of the installation and setup, but I did run into a few more road blocks that fed into the “Best Laid Plans” theme.

One thing that got me was that I had installed the Oracle Management Agent on my PeopleSoft servers before starting to do the research for the PeopleSoft plug-in. Big mistake! I was hoping to monitor the up/down status of the hardware and the available space for the file systems. This worked well, but then I found out that the installation of the agent for the plug-in is required to be owned by, or at least in the same group as, the user that owns the PeopleSoft installation. So after several uninstalls and reinstalls, I was ready to go again. Of course then there was the issue of what user to use when discovering the PeopleSoft targets. The documentation is unclear and I tried several options. Bottom-line is that while the Management Agent can run as a centralized user that shares the group with the PeopleSoft installation, the discovery has to be done by the actual software owner. This was an issue only because we have several environments hosted on the same server in our non-production environments. That was a long paragraph and hopefully makes some sense. If not, I wrote it much better in the paper so you can check it out at Collaborate.

My next challenge had to do with how PeopleSoft identifies its databases. This is something I had no clue about and caught me off-guard. I went through and, with the help of the PeopleSoft Administrators, added all the non-production PeopleSoft targets. One of the cool features of the plug-in is that it allows you to create a system for each environment and a service to measure the login time and make sure the web site is available. When I started to create systems for each environment, Enterprise Manager tried to connect all the components to one database. After further research I found that PeopleSoft stores a GUID in the PSOPTIONS table that needs to be unique for every environment. Since we refresh our test environments with production, all the GUID fields were the same, thus the reason that Enterprise Manager tied everything together. The fix for that is to shut down the application server, set the GUID to a single space in PSOPTIONS, and then restart the application server. The application server then assigns a new GUID and all is wonderful… Unfortunately, the developers, testers, and PeopleSoft administrators are not that happy to give up the environments for outages. Plus, I had to delete all the targets that had been previously discovered and go through the discovery process all over.

At this point I am almost ready to add the production targets to the Enterprise Manager repository and really start using it like it is meant to be used. And of course, I found one more issue. We refreshed one of our non-production environments the other day. Part of the service monitoring includes a user in the PeopleSoft system that only has rights to log in and nothing else. Of course, we decided to use a different password that we could standardize for all non-prod environments, but it is different than what we used the first time. I cannot for the life of me, figure out where to change that password to make the service check work again. I ended up creating a Service Request with Oracle and apparently, they don’t know either. It has been a week and they haven’t given me a response.

Well, I better actually get to finishing the paper. I have not been able to find a lot of information about people using the plug-in so if you are, please drop me a line and let me know. I would love to trade war stories! Also, if you are not already planning on attending Collaborate 09 in Orlando in May, make sure you check it out. It is hard to beat the amount of technical content and the networking opportunities that exist at this one conference.

Until next time, and as always… stay clean!

Upcoming Webinar – Auditing Database Activity for Security and Compliance with Oracle Audit Vault

2009-03-04T10:53:00.000-08:00

In my initial entry, I mentioned that I am involved with the Independent Oracle User Group and the Special Interest Groups. The Enterprise Best Practices SIG is sponsoring a very exciting webinar next week that any Oracle technologist should be participating in. Tanya Baccam is an instructor with the SANS Institute and Roxana Bardescu is a Senior Principal Product Director in the Database Security Marketing Group at Oracle. They are co-presenting a webinar on Audit Vault, Oracle’s new tool for capturing audit information. The webinar is scheduled for Wednesday, March 11 at 2:00 pm Central Time. Here is the abstract:

Using the native auditing capabilities provided by the Oracle Database and Oracle Audit Vault, organizations can easily monitor their databases for suspicious activities and simplify compliance reporting without costly investments in third-party solutions. In this session, you will learn best practices for effectively auditing your Oracle databases as well as how Oracle Audit Vault can proactively collect and analyze that data in real-time to generate alerts and reports that satisfy internal security policies and regulatory compliance.

I have heard both Roxana and Tanya present before and they are very knowledgeable. This should be a very informative webinar. You can register by following this link.

If you want more on security, you can really get an in depth view of database security at Collaborate 09. Tanya will be presenting a University Seminar on Sunday that is an abbreviate version of the SANS Institute’s week long Securing Oracle Database training. This will give you enough information to scare the heck out of your manager!

Check out the webinar… nothing like free training in this day! Happy auditing!

Oracle Releases Grid Control 10g Release 5 – Follow-up

2009-03-04T10:51:00.000-08:00

I just received a follow-up to the webinar yesterday and it is now posted online. You can go to Oracle Technology Network and listen to the recorded webinar here.

After my post yesterday, I received a note on LinkedIn from Rich Schwerin, Senior Product Marketing Manager with the Linux and VM group at Oracle. I have worked with Rich in the past on a roundtable at OpenWorld 2008 and he put me in touch with a couple people for webinars for the IOUG Linux SIG. He reminded me about the new VM features that Oracle put into 10gR5. This is very cool because you can manage the entire stack, right down to the virtual hosts, through the centralized console. Add that to the Linux management pack that is included with your support license for Oracle Enterprise Linux and you really do have a top down management opportunity.

I encourage you to check out the webinar and technical information if you are a VM and/or Enterprise Linux user. The new features should help to make our jobs much easier.

Thanks for the note Rich!

Oracle Releases Grid Control 10g Release 5!

2009-03-03T14:52:00.000-08:00

I know I said my next entry would be about my paper for Collaborate 09, but I sat through Oracle’s webinar today about Grid Control 10gR5 and decided to write about that instead. I’ve been waiting for this release because a lot of the 11g database features have been built into the Enterprise Manager interface. Real Application Testing and Transparent Data Encryption are features that are in the 11g database, but you had to use the local Enterprise Manager to use them. Now they are built into Grid Control. The release number is 10g Release 5, but this was actually supposed to have been the 11g release. Rumor has it, and presenters at conferences have backed this up, that they were ready to call this 11g, but when the BEA purchase went through they decided to hold off on the 11g release until they evaluate revamping Grid Control to use WebLogic instead of Oracle Application Server. We shall see…

The webinar from today is supposed to be posted by tomorrow, but the slides are already there. This really does look like a very nice, feature packed, upgrade. We are on HP-UX/Intanium at my company so I have to wait a bit longer, but if you are using Windows 32 or Linux/Intel, you are good to go. Check out the Enterprise Manager page on OTN and read about the new features. The Real User Experience Insight looks to be a nice feature for measuring performance at the user level. There are accelerators already created for E-Business Suite, Seibel, and PeopleSoft. This should be fun to play with since I support PeopleSoft.

Oracle is building in their acquisitions as fast as they can. One new acquisition that was mentioned on the webinar is mValent. The acquisition was just completed a couple weeks ago and Oracle is already working to build those features into Grid Control. I’m not familiar with the mValent software at all, but apparently it is supposed to make managing complex applications even easier. It should be interesting to see how all these new features for managing complex application environments fit in with already existing plug-ins. As you will see in my later post, I have had several headaches getting the PeopleSoft plug-in implemented and if these tools turn out to be easier, it may be the end of the current plug-in model.

I better get going on my Collaborate paper. Hopefully once I finish it, I can actually write the blog posting about it. Feel free to drop me a line if you like the blog or have some suggestions. Also, I’d love to hear from you if you have had a chance to play with 10gR5!

2009-02-26T17:22:00.001-08:00

Ian Abramson, president of the Independent Oracle User Group, had a very thoughtful blog posting today regarding spending in the current economic times. Companies have to make tough choices when it comes to spending and sometimes it is difficult to make a difference as a "rank and file" employee. One thing to do is make sure you are getting the most out of what they will give you. This includes making the right choices and pushing the right areas for your career.

I have been attending IOUG events since 2000. This was after attending several classes from Oracle, a couple programming classes, a Sun Solaris class, and some project management classes. The interesting thing is that there are sessions on all of these subjects in the one week of Collaborate and after the session you can have an adult beverage (or a soda) with the presenter and ask him/her real world questions. If you are fortunate enough to have a company that still is able to send you to training, think about the best way to spend those dollars. One week on one subject... or one week on about 400 subjects. Check out the session offerings at the Collaborate 09 site and you can even download a pre-written letter to your manager helping you to sell the experience.

My next post is planned for my adventure while writing my paper for Collaborate so stay tuned. Until then... stay clean!

IOUG Relases Security Survey

2009-02-25T10:17:00.000-08:00

Today, the Independent Oracle User Group released the results of a survey conducted of members about their processes for applying Critical Patch Updates. These patches are Oracle’s method to fix any security holes in the software. Others have described in depth the methodology and the findings, so I will include those links at the end. As for what this means to me, it very much fits into how I have been looking at CPU’s since Oracle started releasing them.

Before I talk about my opinions, I need to have a little disclosure. I am a member of the Independent Oracle User Group and was involved in promoting the survey. The comments included in this blog are my opinions and are not a reflection of other members of IOUG. Hopefully that will protect me if I tick someone off!

The CPU process is relatively simple to apply. Shutdown the databases that are running using the affected Oracle software installation, run the patch process, restart the database, and run the .sql script that is delivered. This patch process has been around for a long time for “one-off” patches and the Security team in Oracle has made it relatively fool-proof.

So why, as the survey finds, do people not install these as regularly as Oracle would like? I believe there are a couple reasons and these are somewhat reflected by the answers to the survey. First, everyone is busy keeping their environments running and implementing new projects. Most people are attempting to accomplish more within their organizations with less resources and it is difficult to find the time to perform the patching. Second, Oracle tells us that these patches have been through numerous tests, and I believe they have. Up to this point, however, people have been instructed to test, test, and test again. It is difficult to get through a full test cycle on the patches within the three months between the patch cycles. Finally, when technologists (a fancy word for those of us that are in charge of the patching) approach management about needing down time to apply the patches, management wants to know why and what are they gaining. For the most part, plugging security holes is not high on a manager’s priority list when there is a backlog of projects that need to be implemented.

How do we as technologists work through the listed challenges? I think the most important thing is to make sure you are educated. This can be done by reading Oracle’s readme files and any documentation about Critical Patch Updates. Also, conferences, such as Collaborate 09 have many sessions presented by users and by Oracle regarding the CPU process and security in general. Another point to work on is convincing management that these patches are important and have relatively low risk. We still need to run some testing, but most people will not be required to complete a full blown regression test.

Check out the survey and read what others are saying. The survey can be found here:

http://enterprisesig.oracle.ioug.org

Two other blogs can be found at the following. I will also update as any other news or commentary is posted.

http://blogs.oracle.com/security http://michelledbaunleashed.blogspot.com

The 5 Ws of My Blog

2009-02-25T10:06:00.000-08:00

For my initial blog entry, I decided to ask myself and then answer a few questions. Who am I? Why would I write a blog? What will I write about? Where can you find my interests on the internet? When will I post entries to my blog?

Who Am I?

Ah yes, one of the most asked questions in the universe. My name is Todd Sheetz. I am an Oracle DBA in the Milwaukee, WI area. I currently work for Veolia Environmental services and support a PeopleSoft/Oracle 10g environment. I live, with my wife and two daughters, in Waukesha, a suburb of Milwaukee. I am an active member of the Independent Oracle User Group, but more on that later.

Why would I write a blog?

For lack of a better reason, why not? I figure this will help me to organize thoughts, hopefully start some discussions, and be one more area to promote groups and events that I think are worthwhile.

What will I write about?

For the most part, I plan on writing about projects I am working on and organizations I am involved with. Other “life” topics may find their way here, but those will be few. My interests, work-wise, include Oracle database administration, Enterprise Manager Grid Control, Linux, PeopleSoft administration, and disaster recovery. Besides work, I am a member of the Independent Oracle User Group and have been very involved in the Special Interest Groups. I will be posting updates about the going’s on of those groups also.

Where can you find my interests on the internet?

Besides trying to see how this blogging thing will go, I am on LinkedIn and Facebook. I will have links as I go to www.ioug.org and various SIG events.

When will I post entries to my blog?

Usually, I plan on posting when something interesting piques my interest. There are several webinars coming up that I will be participating in so I will post links and possibly summaries for those. Otherwise, it will just be when I have something that I think others may be interested in.

Until next time… stay clean!