Last week was Collaborate in Las Vegas. I always attempt to keep up with what is going on during the week, but there is always too much to do and not enough time. So here I am a week later putting my thoughts into a blog.
I have been attending IOUG conferences since 1999. The first few were IOUG-A Live events and were great learning experiences for me as I was growing as a DBA. Five years ago, we started co-sponsoring Collaborate with our friends at the OAUG and Quest. As I have gotten more involved in the user group community, the user community and our conferences have taken on a new meaning to me, but at the same time, have become more important.
Conferences have been around for a long time. Collaborate is an excellent opportunity for learning and networking. We say that all year long, but having just been there it still amazes me to see the knowledge level that is walking the halls at this conference. One of the coolest things for me with all the social media and networking going on, is putting a face and a hand shake to a Twitter handle or a Facebook page. These are people we know only 140 characters at a time. Actually meeting and having a face to face conversation is a treat.
My week started with several meetings for the IOUG Board. We welcomed a few new members and are sad to see some great leaders rolling off. This is normal for us, but for me personally, the three guys rolling off, Ian Abramson, Andy Flower, and Kent Hinkley, have all had a huge impact. They are part of the reason I am on the Board and I have learned a lot from all of them.
One highlight of the week for me was hearing Mark Kelly talk about his life and almost losing his wife Gabby to a shooting. His presentation was mesmerizing and had many good points for leadership and life. After the speech, the Boards of the user groups had the privilege of having Mark sign copies of his book for us. I had mine signed to my daughters and Mark reiterated one of his messages to them, "Reach for the stars!"
It is difficult to summarize the amount of technical content that was shared last week. Users, managers, leaders, Oracle technologists all sharing openly of their experiences is a great thing. There were almost 6000 total attendees at Collaborate with over 1000 sessions between the three groups. Add in the face to face networking and the fun parties, Collaborate has turned into one of the highlights of my year, every year.
Next year we will be in Denver. We were there a few years ago and I really liked the convention center there. I will be succeeding Michelle Malcher as Director of Education for Collaborate next year. Michelle has done a great job over the last two years making some great changes and introducing some new programs. We, as a user group, will be revisiting some of our educational content and offerings over the next couple months. We also have an incredible Conference Committee of dedicated, passionate people that I am looking forward to working with. If you have questions about this year's conference or suggestions for next, please let me know. We want to make sure that next year is the best year yet.
Friday, May 4, 2012
Tuesday, September 6, 2011
IOUG Real World Performance - Milwaukee
The IOUG Real World Performance Tour has finally arrived in Milwaukee! I have felt a little like a teenager waiting for my favorite band to come to town. Tom Kyte, Andrew Holdsworth and Graham Wood are here talking about Oracle performance and everyone in the room is listening with rapt attention. So far, we know that most performance problems are the network guys... something we DBA's have known for a long time. In reality, the performance issues come in all over the stack and limiting those become our job. Andrew just loaded 2.5 million rows in a minute and a half. I have already learned a lot about SQL Loader, external tables and statistics.
Much of this information is stuff that Tom, Andrew and Graham have learned over many years of working with the product. There are things here that have been learned through Exadata development and are still valid in the non-Exadata world.
I would highly recommend checking out this program if it is coming to a town near you. If it isn't, and you are interested in having the tour visit your area, let the IOUG know!
Much of this information is stuff that Tom, Andrew and Graham have learned over many years of working with the product. There are things here that have been learned through Exadata development and are still valid in the non-Exadata world.
I would highly recommend checking out this program if it is coming to a town near you. If it isn't, and you are interested in having the tour visit your area, let the IOUG know!
Wednesday, March 23, 2011
Recommendation Pays Off (aka I Told You So!)
Yesterday, Oracle announced that they will no longer be developing for the Itanium processor. SearchOracle posted a blog this morning with a few more details. This caught my attention because last summer my company made the move from HP-UX on Itanium to Oracle Enterprise Linux on X86-64. The initial plan a year ago was to replace the current Itanium server with a bigger one to support our growing processing needs of the Oracle environment. When we started scoping the project, the DBA team suggested looking at Oracle RAC and using smaller, commodity hardware with Linux. Long story short, that was the ultimate decision, and based on the news yesterday, turned out to be the right one. Any time you can make a recommendation that ultimately saves the company time and money, you can call that a success. In this case, if we had moved forward with replacing the existing platform with the same processor, we would be looking for another replacement.
Understanding the technology that you are supporting in your job is just the start. This example shows that keeping an eye on trends and really trying to understand where your company is and where you are going is even more important than being a good technologist. Much of the decision came down to the cost of Itanium, but knowing that the technology did not have the backing that it once did had an impact. Also, Oracle had already put Itanium into its second round anytime releases came out. I was able to learn about those trends while being involved in the Oracle user community. A big part of that is attending conferences like Oracle's OpenWorld and Collaborate which is co-hosted by IOUG, OAUG and Quest. Collaborate 11 will be April 10-14 in Orlando. This is a great way to learn about the latest trends in the industry and learn the latest technology from experts that do it every day. I will be presenting my paper about moving from Itanium to Oracle RAC on Linux. Register today!
Understanding the technology that you are supporting in your job is just the start. This example shows that keeping an eye on trends and really trying to understand where your company is and where you are going is even more important than being a good technologist. Much of the decision came down to the cost of Itanium, but knowing that the technology did not have the backing that it once did had an impact. Also, Oracle had already put Itanium into its second round anytime releases came out. I was able to learn about those trends while being involved in the Oracle user community. A big part of that is attending conferences like Oracle's OpenWorld and Collaborate which is co-hosted by IOUG, OAUG and Quest. Collaborate 11 will be April 10-14 in Orlando. This is a great way to learn about the latest trends in the industry and learn the latest technology from experts that do it every day. I will be presenting my paper about moving from Itanium to Oracle RAC on Linux. Register today!
Tuesday, January 25, 2011
Wealth of Oracle Knowledge
I have been working with Oracle technologies for almost 15 years. One thing that has changed over that time is how we gain knowledge about the technologies. I was fortunate when I graduated from college and my company sent me to several weeks of training. I learned a new programming language and took all the Oracle University classes to become certified in Oracle 7 (yes, Oracle 7).
Today, Oracle University is still going strong, but we also have Google, blogs, Twitter, and, of course, User Groups. With this wealth of knowledge we can find just about anything at any time. The question then becomes, is this the best answer, or just an answer. This is where I feel that the User Groups set ourselves apart. Not only is the information there, but it is being written, presented, and discussed by the thought leaders. IOUG is participating in three events that really underline this point.
As I write today, I am attending the International Oracle User Community User Group Leaders' Summit. This is an event that is sponsored by Oracle and takes place on the Oracle campus in Redwood Shores. This meeting is a mini conference for leaders of the Oracle Community. There is some great information shared by Oracle here, but the bigger reason to attend is to talk with other leaders to find out what they are doing and what has been successful. There is a large focus this year on MySQL and Java given the Sun acquisition that was finalized at this time last year. We are all still learning from each other and it is exciting to get to know two entirely new user communities. I will be taking a lot of this information back to IOUG and working on growing our community, but I will also take some of the new information back to my day job.
The next, very exciting, event that we are involved in was just recently announced. This new event is called "IOUG Presents A Day of Real World Performance with Tom Kyte, Andrew Holdsworth and Graham Wood". This is going to be a full day of performance engineering with the three biggest experts within Oracle. These Oracle experts will debate, discuss and delineate the best practices for designing hardware architectures, deploying Oracle databases, and developing applications that deliver the fastest possible performance for your business. This is going to be an excellent way to learn directly from the experts and have an opportunity to ask direct questions. There are four sessions currently scheduled on the West Coast, but we are working on scheduling more sessions throughout the country. Check out the information Real World Performance Tuning page at ioug.org to see the currently scheduled dates and check back soon for future dates. Also, let me know if you are interested helping with this event in your area and I will get you in touch with our scheduling people.
Finally, our annual conference planning is in full swing for Collaborate 11. IOUG, OAUG and Quest International user communities come together annually to put on the on of the leading, user based, Oracle technology conference in the United States. I am currently working on two papers, including one that will be part of the Oracle RAC Boot Camp. Going back to our roots, you will find a DBA 101 Boot Camp along with BI, Development, Manageability and many others. The other exciting news from Collaborate this year is MySQL. With Oracle's acquisition of Sun last year, the MySQL Community is now part of the Oracle Community. IOUG is offering over 50 sessions just on MySQL. Those of you that are multi database shops can now come to Collaborate and get all your answers. I am excited about being a part of Collaborate again this year. We are going to have our best show ever.
The User Community is busier than ever with more opportunities to learn and get involved. Join us for a Real World Performance session or at Collaborate. I look forward to seeing a lot of people that I see once a year and meeting many more.
Today, Oracle University is still going strong, but we also have Google, blogs, Twitter, and, of course, User Groups. With this wealth of knowledge we can find just about anything at any time. The question then becomes, is this the best answer, or just an answer. This is where I feel that the User Groups set ourselves apart. Not only is the information there, but it is being written, presented, and discussed by the thought leaders. IOUG is participating in three events that really underline this point.
As I write today, I am attending the International Oracle User Community User Group Leaders' Summit. This is an event that is sponsored by Oracle and takes place on the Oracle campus in Redwood Shores. This meeting is a mini conference for leaders of the Oracle Community. There is some great information shared by Oracle here, but the bigger reason to attend is to talk with other leaders to find out what they are doing and what has been successful. There is a large focus this year on MySQL and Java given the Sun acquisition that was finalized at this time last year. We are all still learning from each other and it is exciting to get to know two entirely new user communities. I will be taking a lot of this information back to IOUG and working on growing our community, but I will also take some of the new information back to my day job.
The next, very exciting, event that we are involved in was just recently announced. This new event is called "IOUG Presents A Day of Real World Performance with Tom Kyte, Andrew Holdsworth and Graham Wood". This is going to be a full day of performance engineering with the three biggest experts within Oracle. These Oracle experts will debate, discuss and delineate the best practices for designing hardware architectures, deploying Oracle databases, and developing applications that deliver the fastest possible performance for your business. This is going to be an excellent way to learn directly from the experts and have an opportunity to ask direct questions. There are four sessions currently scheduled on the West Coast, but we are working on scheduling more sessions throughout the country. Check out the information Real World Performance Tuning page at ioug.org to see the currently scheduled dates and check back soon for future dates. Also, let me know if you are interested helping with this event in your area and I will get you in touch with our scheduling people.
Finally, our annual conference planning is in full swing for Collaborate 11. IOUG, OAUG and Quest International user communities come together annually to put on the on of the leading, user based, Oracle technology conference in the United States. I am currently working on two papers, including one that will be part of the Oracle RAC Boot Camp. Going back to our roots, you will find a DBA 101 Boot Camp along with BI, Development, Manageability and many others. The other exciting news from Collaborate this year is MySQL. With Oracle's acquisition of Sun last year, the MySQL Community is now part of the Oracle Community. IOUG is offering over 50 sessions just on MySQL. Those of you that are multi database shops can now come to Collaborate and get all your answers. I am excited about being a part of Collaborate again this year. We are going to have our best show ever.
The User Community is busier than ever with more opportunities to learn and get involved. Join us for a Real World Performance session or at Collaborate. I look forward to seeing a lot of people that I see once a year and meeting many more.
Thursday, October 28, 2010
Oracle Technology Day - Milwaukee
Today I am participating in Oracle Technology Day in Brookfield, WI. Brookfield is a suburb of Milwaukee and closer to my house than work so the commute was better than normal. Those of you that participate in these regularly know that it is an opportunity for Oracle to buy us breakfast and lunch, and in return, they get an opportunity to tell us about their technology. We also get an opportunity to network with several people from the area and find out what others are doing. Meeting others from the are is the best part of these days. The technology discussions are interesting, but most of it is somewhat of a repeat from OpenWorld. So far I have seen about ten people that I have met somewhere in my career history. This is also why I am involved in IOUG because I get this kind of interaction regularly.
Right now is an update on Oracle Technology keynote. We are currently going over the history of the database and leading into where we are now. This is an interesting story to think of going from Oracle 4 back in 1984 to 11gR2 running in a Grid environment now. Add to that where we are going with Cloud Computing and it makes for a very interesting story. Today includes information on Exadata, Security, Fusion Middleware, BI, GoldenGate and much more. I will be jumping around among the three tracks because it all is very interesting and we have things going on in all areas at work and at IOUG. I like to use these days to find out more about what we can be getting to our members. It's funny how all the topics that are covered today are the same topics we hear that our membership is interested in. That is partially due to the User Groups giving Oracle feedback about what our members want. Oracle has done a good job recently with allowing the User Groups to have advocacy back into Oracle and allow us to get the message out. Of course, we still like to talk about what doesn't work and what the products really do which is why we stay independent.
I'm looking forward to hearing more about Fusion Middleware and BI opportunities. Also talking about IT Sprawl and better management of the data center should be good topics. These will be helpful as we grow our current RAC environment and start to consolidate databases onto it. I probably will skip some of the Exadata stuff since Larry talked so much about it at OpenWorld and my current company really doesn't have the volume to warrant that kind of investment. It still sounds very cool though! Again, I'm mostly looking forward to lunch and being able to catch up with people I know and meeting some new ones. Everyone has a different technology story and you never know when you might pick up something you can use. I'm already looking forward to April and Collaborate. That story sharing is a daily thing at Collaborate. Registration for Collaborate opens next month. Make sure to check it out and come share your story. Watch for updates at http://collaborate11.ioug.org.
Hopefully, I'll have some new information to share after Tech day today. I'm sure I will at least have some new contacts and stories!
Right now is an update on Oracle Technology keynote. We are currently going over the history of the database and leading into where we are now. This is an interesting story to think of going from Oracle 4 back in 1984 to 11gR2 running in a Grid environment now. Add to that where we are going with Cloud Computing and it makes for a very interesting story. Today includes information on Exadata, Security, Fusion Middleware, BI, GoldenGate and much more. I will be jumping around among the three tracks because it all is very interesting and we have things going on in all areas at work and at IOUG. I like to use these days to find out more about what we can be getting to our members. It's funny how all the topics that are covered today are the same topics we hear that our membership is interested in. That is partially due to the User Groups giving Oracle feedback about what our members want. Oracle has done a good job recently with allowing the User Groups to have advocacy back into Oracle and allow us to get the message out. Of course, we still like to talk about what doesn't work and what the products really do which is why we stay independent.
I'm looking forward to hearing more about Fusion Middleware and BI opportunities. Also talking about IT Sprawl and better management of the data center should be good topics. These will be helpful as we grow our current RAC environment and start to consolidate databases onto it. I probably will skip some of the Exadata stuff since Larry talked so much about it at OpenWorld and my current company really doesn't have the volume to warrant that kind of investment. It still sounds very cool though! Again, I'm mostly looking forward to lunch and being able to catch up with people I know and meeting some new ones. Everyone has a different technology story and you never know when you might pick up something you can use. I'm already looking forward to April and Collaborate. That story sharing is a daily thing at Collaborate. Registration for Collaborate opens next month. Make sure to check it out and come share your story. Watch for updates at http://collaborate11.ioug.org.
Hopefully, I'll have some new information to share after Tech day today. I'm sure I will at least have some new contacts and stories!
Sunday, September 19, 2010
OpenWorld User Group Forum - Best Day of the Week! #oow10
It has been an exciting and fast paced start to the week. At the moment I am in the User Group Pavilion talking to a lot of people about the benefits of joining a users group. I've actually been trying to get to the post for a couple hours and get a few words in before saying hi to someone.
If you are here, stop by the booth and check us out. I'd love to say hi and learn about what you are doing here. If not, check out www.ioug.org and find out what we are up to. Especially remember about Collaborate 11 in April. This afternoon you can get a sneak peak of the content that you will see at Collaborate during the User Group Forum. All the IOUG sessions are in the 2001-2008 block of rooms in Moscone West.
More later... we have a line!
If you are here, stop by the booth and check us out. I'd love to say hi and learn about what you are doing here. If not, check out www.ioug.org and find out what we are up to. Especially remember about Collaborate 11 in April. This afternoon you can get a sneak peak of the content that you will see at Collaborate during the User Group Forum. All the IOUG sessions are in the 2001-2008 block of rooms in Moscone West.
More later... we have a line!
Thursday, September 9, 2010
Error Stack
Most people think of an error stack as the list of errors that is generated when there is an issue. For example and ORA-00600 causes a few other errors and gives you a stack of errors to unravel. In our case, earlier this week we had a stack of errors that caused a minor database outage. We went live a few weeks ago with Oracle Real Application Clusters to support PeopleSoft Financials. This was a major upgrade from our standalone HP-UX server. Of course there were several things that we learned before going live, but a couple caught us by surprise.
With the previous architecture, we were using Data Guard and replicating to a hot disaster recovery site. Shortly after go-live with RAC, we were working on getting this set up again. I only go into this part because it was one of the things that caused our error stack. In short, the "error stack" is below. I will explain more shortly.
"Error Stack"
1. Production database lost contact with DR
2. No monitoring of DR site being up to date
3. RMAN backups successful, but not deleting logs because they were needed for DR
4. Archive log destination filling up
5. Monitoring was set, but no e-mail or page notification
Error number one that would have avoided our production outage was the logs not reaching the Data Guard server. We are still unsure of why the production site lost contact with DR. The current theory was an incomplete setup of Data Guard Broker. We were received TNS errors in the alert log, but these were not reported in OEM.
Before we went live with the RAC environment, we had Data Guard running for our single instance 10g database and had a script that checked the sequence number between production and DR to make sure we were up to date and would page if we got too far behind. With the implementation of RAC, that script no longer worked and we did not get a chance to fix it when we turned on the Data Guard database. That was error number two that if we had caught, we would not have had an outage.
Our RMAN backups are scripted and we use OEM pretty much only as a job scheduler. This was setup originally because the company did not have OEM up and running, and when we did get it implemented, we didn't want to rely on it for backup/recovery because we did not have a DR plan that included OEM. That way we could just run the scripts for backup/recovery in the event of a DR or an OEM outage. The logs from the scripts are e-mailed to the DBA's, but unless there is a real problem, they come up as successful. In this case, the backups of the archive logs were successful, but our scripts are supposed to delete the logs and that part was failing. We did not know the delete was failing because the overall script and the backup itself was successful. RMAN was not deleting the logs because they could not be transmitted to the DR site and RMAN was trying to protect the DR site.
Since the archive logs were not being deleted as we expected by RMAN, our archive log destination filled up. We still have to figure out what is different between 10g and 11g here and I'm assuming for now that this is a new feature since we didn't really change anything else. Since, we changed our scripts to do force delete on the archive logs because we can always restore them if we need them for DR.
The monitoring piece of the error stack is the one that bugs me the most. I had set the monitoring up via OEM and the alert was actually triggered. The kicker was that I forgot to set the e-mail and pager notifications. If we had happened to be looking at the alert screen on OEM we would have seen it. Obviously, this has been remedied and we are not receiving e-mail messages and pages.
So, there you go. Our version of an error stack. When unraveling all of this, I kept thinking to myself that any one of these would have saved us an outage. Obviously a learning experience, but still frustrating to know we missed that many little things that added up to a big one. Fortunately, we did get notified that the database was unavailable and the total outage was only about 15 minutes. I guess it can always be worse!
Hope to see many people at OpenWorld. One of our PeopleSoft administrators and I are presenting the implementation of our PeopleSoft upgrade and RAC implementation so if you want to chat look me up. I will be posting something more soon about other plans for OpenWorld.
With the previous architecture, we were using Data Guard and replicating to a hot disaster recovery site. Shortly after go-live with RAC, we were working on getting this set up again. I only go into this part because it was one of the things that caused our error stack. In short, the "error stack" is below. I will explain more shortly.
"Error Stack"
1. Production database lost contact with DR
2. No monitoring of DR site being up to date
3. RMAN backups successful, but not deleting logs because they were needed for DR
4. Archive log destination filling up
5. Monitoring was set, but no e-mail or page notification
Error number one that would have avoided our production outage was the logs not reaching the Data Guard server. We are still unsure of why the production site lost contact with DR. The current theory was an incomplete setup of Data Guard Broker. We were received TNS errors in the alert log, but these were not reported in OEM.
Before we went live with the RAC environment, we had Data Guard running for our single instance 10g database and had a script that checked the sequence number between production and DR to make sure we were up to date and would page if we got too far behind. With the implementation of RAC, that script no longer worked and we did not get a chance to fix it when we turned on the Data Guard database. That was error number two that if we had caught, we would not have had an outage.
Our RMAN backups are scripted and we use OEM pretty much only as a job scheduler. This was setup originally because the company did not have OEM up and running, and when we did get it implemented, we didn't want to rely on it for backup/recovery because we did not have a DR plan that included OEM. That way we could just run the scripts for backup/recovery in the event of a DR or an OEM outage. The logs from the scripts are e-mailed to the DBA's, but unless there is a real problem, they come up as successful. In this case, the backups of the archive logs were successful, but our scripts are supposed to delete the logs and that part was failing. We did not know the delete was failing because the overall script and the backup itself was successful. RMAN was not deleting the logs because they could not be transmitted to the DR site and RMAN was trying to protect the DR site.
Since the archive logs were not being deleted as we expected by RMAN, our archive log destination filled up. We still have to figure out what is different between 10g and 11g here and I'm assuming for now that this is a new feature since we didn't really change anything else. Since, we changed our scripts to do force delete on the archive logs because we can always restore them if we need them for DR.
The monitoring piece of the error stack is the one that bugs me the most. I had set the monitoring up via OEM and the alert was actually triggered. The kicker was that I forgot to set the e-mail and pager notifications. If we had happened to be looking at the alert screen on OEM we would have seen it. Obviously, this has been remedied and we are not receiving e-mail messages and pages.
So, there you go. Our version of an error stack. When unraveling all of this, I kept thinking to myself that any one of these would have saved us an outage. Obviously a learning experience, but still frustrating to know we missed that many little things that added up to a big one. Fortunately, we did get notified that the database was unavailable and the total outage was only about 15 minutes. I guess it can always be worse!
Hope to see many people at OpenWorld. One of our PeopleSoft administrators and I are presenting the implementation of our PeopleSoft upgrade and RAC implementation so if you want to chat look me up. I will be posting something more soon about other plans for OpenWorld.
Subscribe to:
Posts (Atom)
