Thursday, October 28, 2010

Oracle Technology Day - Milwaukee

Today I am participating in Oracle Technology Day in Brookfield, WI.  Brookfield is a suburb of Milwaukee and closer to my house than work so the commute was better than normal.  Those of you that participate in these regularly know that it is an opportunity for Oracle to buy us breakfast and lunch, and in return, they get an opportunity to tell us about their technology.  We also get an opportunity to network with several people from the area and find out what others are doing.  Meeting others from the are is the best part of these days.  The technology discussions are interesting, but most of it is somewhat of a repeat from OpenWorld.  So far I have seen about ten people that I have met somewhere in my career history.  This is also why I am involved in IOUG because I get this kind of interaction regularly.

Right now is an update on Oracle Technology keynote.  We are currently going over the history of the database and leading into where we are now.  This is an interesting story to think of going from Oracle 4 back in 1984 to 11gR2 running in a Grid environment now.  Add to that where we are going with Cloud Computing and it makes for a very interesting story.  Today includes information on Exadata, Security, Fusion Middleware, BI, GoldenGate and much more.  I will be jumping around among the three tracks because it all is very interesting and we have things going on in all areas at work and at IOUG.  I like to use these days to find out more about what we can be getting to our members.  It's funny how all the topics that are covered today are the same topics we hear that our membership is interested in.  That is partially due to the User Groups giving Oracle feedback about what our members want.  Oracle has done a good job recently with allowing the User Groups to have advocacy back into Oracle and allow us to get the message out.  Of course, we still like to talk about what doesn't work and what the products really do which is why we stay independent.

I'm looking forward to hearing more about Fusion Middleware and BI opportunities.  Also talking about IT Sprawl and better management of the data center should be good topics.  These will be helpful as we grow our current RAC environment and start to consolidate databases onto it.  I probably will skip some of the Exadata stuff since Larry talked so much about it at OpenWorld and my current company really doesn't have the volume to warrant that kind of investment.  It still sounds very cool though!  Again, I'm mostly looking forward to lunch and being able to catch up with people I know and meeting some new ones.  Everyone has a different technology story and you never know when you might pick up something you can use.  I'm already looking forward to April and Collaborate.  That story sharing is a daily thing at Collaborate.  Registration for Collaborate opens next month.  Make sure to check it out and come share your story.  Watch for updates at

Hopefully, I'll have some new information to share after Tech day today.  I'm sure I will at least have some new contacts and stories!

Sunday, September 19, 2010

OpenWorld User Group Forum - Best Day of the Week! #oow10

It has been an exciting and fast paced start to the week.  At the moment I am in the User Group Pavilion talking to a lot of people about the benefits of joining a users group.  I've actually been trying to get to the post for a couple hours and get a few words in before saying hi to someone.

If you are here, stop by the booth and check us out.  I'd love to say hi and learn about what you are doing here. If not, check out and find out what we are up to.  Especially remember about Collaborate 11 in April.  This afternoon you can get a sneak peak of the content that you will see at Collaborate during the User Group Forum.  All the IOUG sessions are in the 2001-2008 block of rooms in Moscone West.

More later... we have a line!

Thursday, September 9, 2010

Error Stack

Most people think of an error stack as the list of errors that is generated when there is an issue.  For example and ORA-00600 causes a few other errors and gives you a stack of errors to unravel.  In our case, earlier this week we had a stack of errors that caused a minor database outage.  We went live a few weeks ago with Oracle Real Application Clusters to support PeopleSoft Financials.  This was a major upgrade from our standalone HP-UX server.  Of course there were several things that we learned before going live, but a couple caught us by surprise.

With the previous architecture, we were using Data Guard and replicating to a hot disaster recovery site.  Shortly after go-live with RAC, we were working on getting this set up again.  I only go into this part because it was one of the things that caused our error stack.  In short, the "error stack" is below.  I will explain more shortly.

"Error Stack"

1.  Production database lost contact with DR
2.  No monitoring of DR site being up to date
3.  RMAN backups successful, but not deleting logs because they were needed for DR
4.  Archive log destination filling up
5.  Monitoring was set, but no e-mail or page notification

Error number one that would have avoided our production outage was the logs not reaching the Data Guard server.  We are still unsure of why the production site lost contact with DR.  The current theory was an incomplete setup of Data Guard Broker.  We were received TNS errors in the alert log, but these were not reported in OEM.

Before we went live with the RAC environment, we had Data Guard running for our single instance 10g database and had a script that checked the sequence number between production and DR to make sure we were up to date and would page if we got too far behind.  With the implementation of RAC, that script no longer worked and we did not get a chance to fix it when we turned on the Data Guard database.  That was error number two that if we had caught, we would not have had an outage.

Our RMAN backups are scripted and we use OEM pretty much only as a job scheduler.  This was setup originally because the company did not have OEM up and running, and when we did get it implemented, we didn't want to rely on it for backup/recovery because we did not have a DR plan that included OEM.  That way we could just run the scripts for backup/recovery in the event of a DR or an OEM outage.  The logs from the scripts are e-mailed to the DBA's, but unless there is a real problem, they come up as successful.  In this case, the backups of the archive logs were successful, but our scripts are supposed to delete the logs and that part was failing.  We did not know the delete was failing because the overall script and the backup itself was successful.  RMAN was not deleting the logs because they could not be transmitted to the DR site and RMAN was trying to protect the DR site.

Since the archive logs were not being deleted as we expected by RMAN, our archive log destination filled up.    We still have to figure out what is different between 10g and 11g here and I'm assuming for now that this is a new feature since we didn't really change anything else.  Since, we changed our scripts to do force delete on the archive logs because we can always restore them if we need them for DR.

The monitoring piece of the error stack is the one that bugs me the most.  I had set the monitoring up via OEM and the alert was actually triggered.  The kicker was that I forgot to set the e-mail and pager notifications.  If we had happened to be looking at the alert screen on OEM we would have seen it.  Obviously, this has been remedied and we are not receiving e-mail messages and pages.

So, there you go.  Our version of an error stack.  When unraveling all of this, I kept thinking to myself that any one of these would have saved us an outage.  Obviously a learning experience, but still frustrating to know we missed that many little things that added up to a big one.  Fortunately, we did get notified that the database was unavailable and the total outage was only about 15 minutes.  I guess it can always be worse!

Hope to see many people at OpenWorld.  One of our PeopleSoft administrators and I are presenting the implementation of our PeopleSoft upgrade and RAC implementation so if you want to chat look me up.  I will be posting something more soon about other plans for OpenWorld.

Tuesday, July 27, 2010

My OpenWorld 2010 Planning

I am supposed to be working on the next post for my PeopleSoft/RAC project, but decided to take a break from that and talk about OpenWorld.  As just about anyone in the Oracle universe is aware, OpenWorld is September 19th to 23rd.  This year I am doing three sessions along with helping to organize the User Group Forum on Sunday.  Before that all gets going though, I'm hoping to catch the Milwaukee Brewers at the San Francisco Giants.  I am a huge Brewers fan and I have never seen a game outside of Milwaukee.  The Brewers are in San Francisco for a weekend series and I'm hoping to get to the game on Saturday evening.  Anyone want to join me?

Sunday is again User Group Forum day at OpenWorld.  The schedule is a little different and the sessions are not starting until 12:30 do to travel restrictions on Satruday because of Yom Kippur.  IOUG again has eight rooms that we have schedule four sessions per room.  In addition, Oracle is having MySQL Sunday during the same time at the Marriott.  IOUG will be participating with some sessions there too.  This will split the audience a little bit on Sunday, but we are hoping for huge crowds at both events.  Check out the IOUG list of sessions or via Schedule Builder or Content Catalog on the OpenWorld site.

My personal schedule is already busy as usual.  I will be helping out at the IOUG booth again along with various meetings.  Plus, I am involved with three sessions this year.  For the User Group Forum on Sunday, I will be leading a round table discussion about Linux.  We will have several users as well as people from the Oracle development team.  This has been a very interesting and well attended session in years past.  People have had many questions and there is a lot of information sharing that takes place.  If you are an Oracle user that has anything installed on Linux, this should be a very informational sessions.  Come and learn about Linux and also share your experiences.

Recently, my company updated our PeopleSoft application.  As our PeopleSoft Administrator (PSA) likes to say, "we did a minor upgrade."  That "minor" upgrade included PeopleTools 8.49 to 8.50.08, Application Bundles from 25 through 28, and implementing Multi-Language.  We are toward the end of phase 2 of the project, which is to move to Oracle Real Application Clusters (RAC) for the database.  The PSA that was in charge of phase 1 and I are doing a presentation on the overall project.  This should be fun because we should be crossing a large number of functions and users.  The title is more about PeopleTools 8.50 which is fine because I think it will draw a big crowd.

When we decided to replace our PeopleSoft database server we went through an evaluation process to determine the best solution.  As part of that evaluation, we decided to use Linux as our database operating system.  I have been a Linux person for several years and had to convert our HP-UX system administrator into an accepter of Linux.  As part of that conversion, he and I will be doing a Linux session with the Linux product team from Oracle.  They will lead in with how the support model is supposed to work and what is included.  We will finish up with how it all really works.

That's my current wrap up for OpenWorld.  I'm hoping to see a lot of people that I only get to see a couple times a year.  I also get to meet new people every year and it blows my mind how many people are out there that are doing some really cool stuff.  If you are planning on attending, let me know and check out the User Group Forum on Sunday!

Saturday, July 10, 2010

Oracle RAC For PeopleSoft - Part 1

So it's been several months since my last blog entry.  Things have been crazy at work, at home, and with IOUG and unfortunately, blogging is somewhat low on the priority list.  So, what have I been up to?  Well, the home life  would be enough for its own blog with two kids that are very involved in sports.  That is another story though.  Work has been very interesting with a couple projects.  The one I am in the middle of now is implementing Oracle Real Application Clusters (RAC) to support our PeopleSoft database.

I have been a DBA for about 13 years.  It is almost crazy to say that, but okay, I accept it.  I am on my second time around supporting a PeopleSoft environment.  I work for a company that is made up of many divisions and is part of a larger organization made up of several companies.  We have been consolidating financial systems into a PeopleSoft 9.0 Financials system from our divisions, and recently, from one of our sister companies.  Not only is it a challenge to integrate business processes between divisions and companies, but technically, we have been challenged to find a solution that will grow as we continue to consolidate.  The biggest part of that challenge is that we are not sure, from an IT perspective, when we will be consolidating.  We do not want to oversize our architecture now for where we are, but do not want to  undersize for future growth.

Performance of the PeopleSoft database has been questionable since I started at this company two years ago.  We have done quite a bit with indexes, changing some business processes, and actually changing some of the application processes.  A couple big wins involved converting PeopleSoft "temp" tables to Oracle temporary tables.  Even with all the tuning successes, adding many users through growth and consolidation has made it very apparent that we have outgrown our database environment.  This is really not bad considering the environment was implemented over three years ago with limited information.

Given that we had exhausted most of our tuning options and we knew we were bringing on new projects and users, we decided that new hardware was required.  The decision came down to a new, dedicated database server or Real Application Clusters.  The company is still working to consolidate IT operations, so we are not sure what divisions or how many users we will be supporting over the next three to five years.  Since we did not want to suggest spending several hundred thousand dollars on a new server this year, only to have to do it again in a year if we consolidated with another large division, RAC was the logical choice.

This entry has been in the works for several weeks and is getting long enough.  In the coming entries I will talk about how we put together a proof of concept, designed the architecture, built the environment and how the implementation is going.  Stay tuned.  I will make sure the follow-up entries are faster in coming than this one...

Tuesday, April 6, 2010

Collaborate 10 Less than Two Weeks! #c10

With Collaborate 10 less than two weeks away, I realized I have not written much lately.  This year's show promises to be better than ever and I am really excited for some of the new changes along with the usual networking and learning opportunities.

About a week ago, Oracle announced that the Enterprise Manager 11g Launch Event will be taking place on Thursday during Collaborate.  Working with the Enterprise Manager team and the fantastic IOUG staff, we have secured a room and will be carrying the Launch Event live as a Collaborate session.  The event is scheduled to be from 10:00 until noon on Thursday morning, April 22 Las Vegas time.  This works out perfectly for the final two sessions on Thursday before the closing general session.  The Launch Event should be listed in the IOUG and Joint schedulers soon so make sure to get it on your calendar for the week.

For the IOUG Regional Users Group leaders we will be having two meetings that should give you a new opportunity for networking.  Tuesday morning for breakfast (I know it's early, but at least it's not after the party) we will be having a joint meeting with the Quest International User Group RUGs.  This will be an opportunity for the RUG leaders from both organizations to meet and discuss what has worked and not worked for the groups.  Also, we are hoping this might promote some co-location of meetings.  The other opportunity is for the RUG leaders to meet with Oracle field marketing and get to know some of the points of contact in their area.  Dan Goldstein, Senior Director for Oracle Field Marketing, will have his team at Collaborate.  We have set up a lunch meeting for th RUG leaders to meet with the point of contact in the area.  If you area  RUG leader, make sure to check these out and make use of the networking opportunities.

The IOUG Special Interest Groups are also taking a large role again this year.  Many of the SIGs are having annual meetings and round table discussions.  Some of the SIG leaders also participated in creating the content for the Boot Camps, but more on those in a bit.  If you are interested in the user group experience and want to get involved, the SIGs are a great place to start.  Find a topic you are interested in, look up that SIG meeting and join in.  You have nothing to lose and so much to gain.  Enterprise Best Practices, Linux, RAC, BI, Content Management and more are all available.  New this year is the zLinux SIG.  This group used to be the z/OS SIG and have joined forces with the IOUG.  The are also having their own boot camp at Collaborate 10.

Several other new options should prove to make the conference experience even better this year.  If you can't make it for the entire week, check out the two day registration option.  This allows you to take advantage of two days of educational content and pay less than half the cost of the entire conference.  Another option this year is the introduction of Boot Camp style sessions.  Several topics were chosen and all the content for that topic was centralized into a one or two day schedule.  Need to learn about RAC?  Check out the sessions in the two day RAC Boot Camp.  Most sessions are in the same room, so you can just camp out there and learn all you need to know.

Business Intelligence has become one of the hottest areas of IT in a long time.  Get Analytical with BIWA Training Days is our conference within a conference for all things BI.  Anything you wanted to know, and some things you didn't want to know, are all in one location.

Add all of this to what Quest and OAUG are offering and this is the most complete conference around for Oracle technology by users.  Check it out and register here....

I hope to see you in Vegas!  If you are there, make sure to look me up and say hi!

Wednesday, February 10, 2010

Oracle SQL Developer Book Review

I have been asked to review the new Oracle SQL Developer 2.1 book by Sue Harper.  When I was asked, I had to clarify that I was not a SQL Developer user.  I'm an old school DBA that, for the most part, used scripts and SQL Plus.  Developers are always bugging me about why I don't use SQL Developer or TOAD or something like that, but I have my scripts and I like my scripts.  When I accepted the offer to review the book, it was partially so I could learn more about the tool and see if it could help me do my job better.

The book is published by PACKT Publishing in Birmingham, UK.  They sent me an electronic copy of the book to review.  If you would like a sneak peak, you can check it out at the publisher's web site here.  I actually like the electronic version.  It is much easier to carry than an almost 500 page book and makes it easy to switch between SQL Developer and the book.  The whole electronic book thing has intrigued me and I have been looking at the Kindle and Nook, but that is information for a different post.  The electronic version did come in handy as I started my review on the plane and having my laptop on the tray table took up enough space.  I would have had to ask my neighbor to hold the actual book for me if I had a hard copy.

I have started working through the book and when I say working through, I mean working through.  Having never actually used SQL Developer before, I am working through all the examples because I would like to learn as I go.  I'm sure as I get a little deeper into the book, I will be able to read/review a little faster, but right now, it is kind of fun to go through and play with things.  Sue (the author) starts off the book specifically saying that it is not a book to teach SQL, PL/SQL, or Oracle concepts.  There is actually a comment that I found was directed at me (even though she doesn't know me).  Even though I am a command line person, this book is written to get me on the road to a "GUI life".  I like that term and I'm interested in seeing what I can do that may make my day-to-day job easier or faster.

So far, the book has lived up to the introduction.  There have been a couple instances that I felt if you don't know Oracle databases, you might be a little confused, but for the most part I have been able to get started pretty quickly.  I like the object browsing and editing capabilities of the tool and the introduction to them in the book.  The connection handling is also introduced very well.  One thing I didn't have when I started was a pre-built database with the example schemas.  I was able to create an 11g database on my laptop and included the example schemas.  If you are planning on using this book, the example schemas are almost a must.  All the examples are based on the example schemas and they have been very nice to have while going through the book.

I will post further updates as I get farther into the book.  I'm not much of a PL/SQL developer, but I will do my best as I get into the development chapters.  In the meantime, I recommend checking out the sample that is linked above.

Wednesday, January 27, 2010

User Group Leaders' Summit Day 1 - #oraclesun

I was fortunate enough to be invited to the Oracle User Group Leaders' Summit at Oracle HQ in Redwood City this week.  Once the Oracle-Sun deal was given the go ahead by the EC last week, we were invited to sit in live for the big announcement.  My day started early after about five hours of sleep because my flight was late getting in last night.  I met some of my fellow IOUG Board members for breakfast and ran into many others including Ronan Miles, UKOUG Chairman, who went to Brussels with IOUG President, Ian Abramson, to give the User Group position on the acquisition.  Also ran into ODTUG President, Mike Riley, Quest President, Sue Shaw, and OAUG President, Ray Payne.  That sounds like a lot of name dropping, and it is!

From breakfast, we walked over to the Oracle Conference Center and were ushered into the Auditorium.  You can check out all my tweets from the session, but to sum it up, they are pushing hard on the integrated solution, complete platform for applications.  We heard from Charles Phillips, who hoped we had a less eventful week than he has...  Charles also talked about hiring 2000 people and if the current US Administration is looking for a jobs program, "her we are!"  I won't go into politics here, but these are the kind of of things that should be focused on and learned from.

John Fowler, who is now EVP of Hardware Technologies at Oracle discussed the investments in Solaris and Sun hardware moving forward.  Mike Splain, who seems to be the real techie, outlined the accelerated road maps for UltraSPARC and SPARC64.  Oracle is putting more money into these platforms and they are hiring here too!  My question continues to be how many people are getting laid off at the same time that they are hiring all these new positions.  I'm sure the layoffs will get some money to walk away, but are they really just shifting positions?  That hasn't been asked or answered yet.

The key theme continues to be seamless integrations between existing Oracle technologies and the newly acquired server, storage, and operating system.  Supposedly all this testing has been done for many of the Oracle applications.  Can they get these all working as they are advertising with flash technologies, new storage options, and newer Sun hardware and Solaris all remains to be seen.  They keep selling the fact that they are being built and integrated together and if you have issues, you only have one support organization to deal with. 

I'm still in the Oracle/Sun session and I'm sure there will be further information later.  I also get a tour of the Usability Lab this afternoon which should be pretty interesting.  Tomorrow starts the actual meetings including my first OpenWorld 10 planning meeting.  I really wish we could finish one conference before starting planning for the next.  Speaking of Conferences, don't forget about Collaborate 10! There will be sessions covering some of the new Sun technologies!

Wednesday, January 20, 2010

Stay Competitive: Join Me at the COLLABORATE 10 – IOUG Forum

I invite you to join me at COLLABORATE 10 – IOUG Forum, April 18-22, 2010 in Las Vegas, NV.  As a member of the IOUG Board of Directors, I am excited to let you know about the educational content that the conference committee has created for the Forum.

You will have the opportunity to attend educational sessions in all of the tracks offered at COLLABORATE,  including: Application Strategy and Services, Infrastructure Technologies BI/Data Warehousing/EPM, Database, Development, Governance Risk and Compliance, Master Data Management/Application Integration and Professional Development.

The IOUG Forum does not only offer top-notch educational sessions that cover a variety of topics, but it’s also a great place to network with other Oracle professionals.  We are offering many exciting networking opportunities this year. Whether it’s a technical talk during a Birds of a Feather meeting or an informal conversation at the IOUG Welcome Event, the IOUG Forum is the place to meet contacts who can share solutions to your most pressing business questions.

The IOUG will not be the only user group present at the event. For the fifth consecutive year, the Oracle Applications Users Group (OAUG) and Quest International Users Group (Quest) will be co-hosting this event, which means your registration through the IOUG also allows you to attend education sessions offered by these groups.

The main reason I attend is for the networking opportunities with my peers.  In all the years I have been attending IOUG events, the opportunity to discuss current technologies and best practices has been the best way I have found to keep myself current.  The technical content is also amazing.  These are people that are using the technology on a daily basis.

This isn’t some huge vendor show full of sales presentations…it’s presented by users like you and me. If you have been to IOUG events before, you know that we feature real-world education and best practices. There is far too much content to completely list, but you can visit the IOUG Forum Web site to view the most up-to-date education details.

Make sure to register to attend and join me at the IOUG Forum! Please visit the IOUG Web site for further details and to register. Be sure to register by March 18 to save $400 USD off the onsite rate, and choose priority code PC03 during the demographic section (step 4) of the registration process and gain access to a number of IOUG-exclusive offerings. Help me spread the word and send this information to your colleagues and friends who are Oracle users.

Please contact me or the IOUG headquarters at if you have any questions.

I hope to see you in Vegas!