I am supposed to be working on the next post for my PeopleSoft/RAC project, but decided to take a break from that and talk about OpenWorld. As just about anyone in the Oracle universe is aware, OpenWorld is September 19th to 23rd. This year I am doing three sessions along with helping to organize the User Group Forum on Sunday. Before that all gets going though, I'm hoping to catch the Milwaukee Brewers at the San Francisco Giants. I am a huge Brewers fan and I have never seen a game outside of Milwaukee. The Brewers are in San Francisco for a weekend series and I'm hoping to get to the game on Saturday evening. Anyone want to join me?
Sunday is again User Group Forum day at OpenWorld. The schedule is a little different and the sessions are not starting until 12:30 do to travel restrictions on Satruday because of Yom Kippur. IOUG again has eight rooms that we have schedule four sessions per room. In addition, Oracle is having MySQL Sunday during the same time at the Marriott. IOUG will be participating with some sessions there too. This will split the audience a little bit on Sunday, but we are hoping for huge crowds at both events. Check out the IOUG list of sessions or via Schedule Builder or Content Catalog on the OpenWorld site.
My personal schedule is already busy as usual. I will be helping out at the IOUG booth again along with various meetings. Plus, I am involved with three sessions this year. For the User Group Forum on Sunday, I will be leading a round table discussion about Linux. We will have several users as well as people from the Oracle development team. This has been a very interesting and well attended session in years past. People have had many questions and there is a lot of information sharing that takes place. If you are an Oracle user that has anything installed on Linux, this should be a very informational sessions. Come and learn about Linux and also share your experiences.
Recently, my company updated our PeopleSoft application. As our PeopleSoft Administrator (PSA) likes to say, "we did a minor upgrade." That "minor" upgrade included PeopleTools 8.49 to 8.50.08, Application Bundles from 25 through 28, and implementing Multi-Language. We are toward the end of phase 2 of the project, which is to move to Oracle Real Application Clusters (RAC) for the database. The PSA that was in charge of phase 1 and I are doing a presentation on the overall project. This should be fun because we should be crossing a large number of functions and users. The title is more about PeopleTools 8.50 which is fine because I think it will draw a big crowd.
When we decided to replace our PeopleSoft database server we went through an evaluation process to determine the best solution. As part of that evaluation, we decided to use Linux as our database operating system. I have been a Linux person for several years and had to convert our HP-UX system administrator into an accepter of Linux. As part of that conversion, he and I will be doing a Linux session with the Linux product team from Oracle. They will lead in with how the support model is supposed to work and what is included. We will finish up with how it all really works.
That's my current wrap up for OpenWorld. I'm hoping to see a lot of people that I only get to see a couple times a year. I also get to meet new people every year and it blows my mind how many people are out there that are doing some really cool stuff. If you are planning on attending, let me know and check out the User Group Forum on Sunday!
Tuesday, July 27, 2010
Saturday, July 10, 2010
Oracle RAC For PeopleSoft - Part 1
So it's been several months since my last blog entry. Things have been crazy at work, at home, and with IOUG and unfortunately, blogging is somewhat low on the priority list. So, what have I been up to? Well, the home life would be enough for its own blog with two kids that are very involved in sports. That is another story though. Work has been very interesting with a couple projects. The one I am in the middle of now is implementing Oracle Real Application Clusters (RAC) to support our PeopleSoft database.
I have been a DBA for about 13 years. It is almost crazy to say that, but okay, I accept it. I am on my second time around supporting a PeopleSoft environment. I work for a company that is made up of many divisions and is part of a larger organization made up of several companies. We have been consolidating financial systems into a PeopleSoft 9.0 Financials system from our divisions, and recently, from one of our sister companies. Not only is it a challenge to integrate business processes between divisions and companies, but technically, we have been challenged to find a solution that will grow as we continue to consolidate. The biggest part of that challenge is that we are not sure, from an IT perspective, when we will be consolidating. We do not want to oversize our architecture now for where we are, but do not want to undersize for future growth.
Performance of the PeopleSoft database has been questionable since I started at this company two years ago. We have done quite a bit with indexes, changing some business processes, and actually changing some of the application processes. A couple big wins involved converting PeopleSoft "temp" tables to Oracle temporary tables. Even with all the tuning successes, adding many users through growth and consolidation has made it very apparent that we have outgrown our database environment. This is really not bad considering the environment was implemented over three years ago with limited information.
Given that we had exhausted most of our tuning options and we knew we were bringing on new projects and users, we decided that new hardware was required. The decision came down to a new, dedicated database server or Real Application Clusters. The company is still working to consolidate IT operations, so we are not sure what divisions or how many users we will be supporting over the next three to five years. Since we did not want to suggest spending several hundred thousand dollars on a new server this year, only to have to do it again in a year if we consolidated with another large division, RAC was the logical choice.
This entry has been in the works for several weeks and is getting long enough. In the coming entries I will talk about how we put together a proof of concept, designed the architecture, built the environment and how the implementation is going. Stay tuned. I will make sure the follow-up entries are faster in coming than this one...
I have been a DBA for about 13 years. It is almost crazy to say that, but okay, I accept it. I am on my second time around supporting a PeopleSoft environment. I work for a company that is made up of many divisions and is part of a larger organization made up of several companies. We have been consolidating financial systems into a PeopleSoft 9.0 Financials system from our divisions, and recently, from one of our sister companies. Not only is it a challenge to integrate business processes between divisions and companies, but technically, we have been challenged to find a solution that will grow as we continue to consolidate. The biggest part of that challenge is that we are not sure, from an IT perspective, when we will be consolidating. We do not want to oversize our architecture now for where we are, but do not want to undersize for future growth.
Performance of the PeopleSoft database has been questionable since I started at this company two years ago. We have done quite a bit with indexes, changing some business processes, and actually changing some of the application processes. A couple big wins involved converting PeopleSoft "temp" tables to Oracle temporary tables. Even with all the tuning successes, adding many users through growth and consolidation has made it very apparent that we have outgrown our database environment. This is really not bad considering the environment was implemented over three years ago with limited information.
Given that we had exhausted most of our tuning options and we knew we were bringing on new projects and users, we decided that new hardware was required. The decision came down to a new, dedicated database server or Real Application Clusters. The company is still working to consolidate IT operations, so we are not sure what divisions or how many users we will be supporting over the next three to five years. Since we did not want to suggest spending several hundred thousand dollars on a new server this year, only to have to do it again in a year if we consolidated with another large division, RAC was the logical choice.
This entry has been in the works for several weeks and is getting long enough. In the coming entries I will talk about how we put together a proof of concept, designed the architecture, built the environment and how the implementation is going. Stay tuned. I will make sure the follow-up entries are faster in coming than this one...
Tuesday, April 6, 2010
Collaborate 10 Less than Two Weeks! #c10
With Collaborate 10 less than two weeks away, I realized I have not written much lately. This year's show promises to be better than ever and I am really excited for some of the new changes along with the usual networking and learning opportunities.
About a week ago, Oracle announced that the Enterprise Manager 11g Launch Event will be taking place on Thursday during Collaborate. Working with the Enterprise Manager team and the fantastic IOUG staff, we have secured a room and will be carrying the Launch Event live as a Collaborate session. The event is scheduled to be from 10:00 until noon on Thursday morning, April 22 Las Vegas time. This works out perfectly for the final two sessions on Thursday before the closing general session. The Launch Event should be listed in the IOUG and Joint schedulers soon so make sure to get it on your calendar for the week.
For the IOUG Regional Users Group leaders we will be having two meetings that should give you a new opportunity for networking. Tuesday morning for breakfast (I know it's early, but at least it's not after the party) we will be having a joint meeting with the Quest International User Group RUGs. This will be an opportunity for the RUG leaders from both organizations to meet and discuss what has worked and not worked for the groups. Also, we are hoping this might promote some co-location of meetings. The other opportunity is for the RUG leaders to meet with Oracle field marketing and get to know some of the points of contact in their area. Dan Goldstein, Senior Director for Oracle Field Marketing, will have his team at Collaborate. We have set up a lunch meeting for th RUG leaders to meet with the point of contact in the area. If you area RUG leader, make sure to check these out and make use of the networking opportunities.
The IOUG Special Interest Groups are also taking a large role again this year. Many of the SIGs are having annual meetings and round table discussions. Some of the SIG leaders also participated in creating the content for the Boot Camps, but more on those in a bit. If you are interested in the user group experience and want to get involved, the SIGs are a great place to start. Find a topic you are interested in, look up that SIG meeting and join in. You have nothing to lose and so much to gain. Enterprise Best Practices, Linux, RAC, BI, Content Management and more are all available. New this year is the zLinux SIG. This group used to be the z/OS SIG and have joined forces with the IOUG. The are also having their own boot camp at Collaborate 10.
Several other new options should prove to make the conference experience even better this year. If you can't make it for the entire week, check out the two day registration option. This allows you to take advantage of two days of educational content and pay less than half the cost of the entire conference. Another option this year is the introduction of Boot Camp style sessions. Several topics were chosen and all the content for that topic was centralized into a one or two day schedule. Need to learn about RAC? Check out the sessions in the two day RAC Boot Camp. Most sessions are in the same room, so you can just camp out there and learn all you need to know.
Business Intelligence has become one of the hottest areas of IT in a long time. Get Analytical with BIWA Training Days is our conference within a conference for all things BI. Anything you wanted to know, and some things you didn't want to know, are all in one location.
Add all of this to what Quest and OAUG are offering and this is the most complete conference around for Oracle technology by users. Check it out and register here.... http://collaborate10.ioug.org
I hope to see you in Vegas! If you are there, make sure to look me up and say hi!
About a week ago, Oracle announced that the Enterprise Manager 11g Launch Event will be taking place on Thursday during Collaborate. Working with the Enterprise Manager team and the fantastic IOUG staff, we have secured a room and will be carrying the Launch Event live as a Collaborate session. The event is scheduled to be from 10:00 until noon on Thursday morning, April 22 Las Vegas time. This works out perfectly for the final two sessions on Thursday before the closing general session. The Launch Event should be listed in the IOUG and Joint schedulers soon so make sure to get it on your calendar for the week.
For the IOUG Regional Users Group leaders we will be having two meetings that should give you a new opportunity for networking. Tuesday morning for breakfast (I know it's early, but at least it's not after the party) we will be having a joint meeting with the Quest International User Group RUGs. This will be an opportunity for the RUG leaders from both organizations to meet and discuss what has worked and not worked for the groups. Also, we are hoping this might promote some co-location of meetings. The other opportunity is for the RUG leaders to meet with Oracle field marketing and get to know some of the points of contact in their area. Dan Goldstein, Senior Director for Oracle Field Marketing, will have his team at Collaborate. We have set up a lunch meeting for th RUG leaders to meet with the point of contact in the area. If you area RUG leader, make sure to check these out and make use of the networking opportunities.
The IOUG Special Interest Groups are also taking a large role again this year. Many of the SIGs are having annual meetings and round table discussions. Some of the SIG leaders also participated in creating the content for the Boot Camps, but more on those in a bit. If you are interested in the user group experience and want to get involved, the SIGs are a great place to start. Find a topic you are interested in, look up that SIG meeting and join in. You have nothing to lose and so much to gain. Enterprise Best Practices, Linux, RAC, BI, Content Management and more are all available. New this year is the zLinux SIG. This group used to be the z/OS SIG and have joined forces with the IOUG. The are also having their own boot camp at Collaborate 10.
Several other new options should prove to make the conference experience even better this year. If you can't make it for the entire week, check out the two day registration option. This allows you to take advantage of two days of educational content and pay less than half the cost of the entire conference. Another option this year is the introduction of Boot Camp style sessions. Several topics were chosen and all the content for that topic was centralized into a one or two day schedule. Need to learn about RAC? Check out the sessions in the two day RAC Boot Camp. Most sessions are in the same room, so you can just camp out there and learn all you need to know.
Business Intelligence has become one of the hottest areas of IT in a long time. Get Analytical with BIWA Training Days is our conference within a conference for all things BI. Anything you wanted to know, and some things you didn't want to know, are all in one location.
Add all of this to what Quest and OAUG are offering and this is the most complete conference around for Oracle technology by users. Check it out and register here.... http://collaborate10.ioug.org
I hope to see you in Vegas! If you are there, make sure to look me up and say hi!
Wednesday, February 10, 2010
Oracle SQL Developer Book Review
I have been asked to review the new Oracle SQL Developer 2.1 book by Sue Harper. When I was asked, I had to clarify that I was not a SQL Developer user. I'm an old school DBA that, for the most part, used scripts and SQL Plus. Developers are always bugging me about why I don't use SQL Developer or TOAD or something like that, but I have my scripts and I like my scripts. When I accepted the offer to review the book, it was partially so I could learn more about the tool and see if it could help me do my job better.
The book is published by PACKT Publishing in Birmingham, UK. They sent me an electronic copy of the book to review. If you would like a sneak peak, you can check it out at the publisher's web site here. I actually like the electronic version. It is much easier to carry than an almost 500 page book and makes it easy to switch between SQL Developer and the book. The whole electronic book thing has intrigued me and I have been looking at the Kindle and Nook, but that is information for a different post. The electronic version did come in handy as I started my review on the plane and having my laptop on the tray table took up enough space. I would have had to ask my neighbor to hold the actual book for me if I had a hard copy.
I have started working through the book and when I say working through, I mean working through. Having never actually used SQL Developer before, I am working through all the examples because I would like to learn as I go. I'm sure as I get a little deeper into the book, I will be able to read/review a little faster, but right now, it is kind of fun to go through and play with things. Sue (the author) starts off the book specifically saying that it is not a book to teach SQL, PL/SQL, or Oracle concepts. There is actually a comment that I found was directed at me (even though she doesn't know me). Even though I am a command line person, this book is written to get me on the road to a "GUI life". I like that term and I'm interested in seeing what I can do that may make my day-to-day job easier or faster.
So far, the book has lived up to the introduction. There have been a couple instances that I felt if you don't know Oracle databases, you might be a little confused, but for the most part I have been able to get started pretty quickly. I like the object browsing and editing capabilities of the tool and the introduction to them in the book. The connection handling is also introduced very well. One thing I didn't have when I started was a pre-built database with the example schemas. I was able to create an 11g database on my laptop and included the example schemas. If you are planning on using this book, the example schemas are almost a must. All the examples are based on the example schemas and they have been very nice to have while going through the book.
I will post further updates as I get farther into the book. I'm not much of a PL/SQL developer, but I will do my best as I get into the development chapters. In the meantime, I recommend checking out the sample that is linked above.
The book is published by PACKT Publishing in Birmingham, UK. They sent me an electronic copy of the book to review. If you would like a sneak peak, you can check it out at the publisher's web site here. I actually like the electronic version. It is much easier to carry than an almost 500 page book and makes it easy to switch between SQL Developer and the book. The whole electronic book thing has intrigued me and I have been looking at the Kindle and Nook, but that is information for a different post. The electronic version did come in handy as I started my review on the plane and having my laptop on the tray table took up enough space. I would have had to ask my neighbor to hold the actual book for me if I had a hard copy.
I have started working through the book and when I say working through, I mean working through. Having never actually used SQL Developer before, I am working through all the examples because I would like to learn as I go. I'm sure as I get a little deeper into the book, I will be able to read/review a little faster, but right now, it is kind of fun to go through and play with things. Sue (the author) starts off the book specifically saying that it is not a book to teach SQL, PL/SQL, or Oracle concepts. There is actually a comment that I found was directed at me (even though she doesn't know me). Even though I am a command line person, this book is written to get me on the road to a "GUI life". I like that term and I'm interested in seeing what I can do that may make my day-to-day job easier or faster.
So far, the book has lived up to the introduction. There have been a couple instances that I felt if you don't know Oracle databases, you might be a little confused, but for the most part I have been able to get started pretty quickly. I like the object browsing and editing capabilities of the tool and the introduction to them in the book. The connection handling is also introduced very well. One thing I didn't have when I started was a pre-built database with the example schemas. I was able to create an 11g database on my laptop and included the example schemas. If you are planning on using this book, the example schemas are almost a must. All the examples are based on the example schemas and they have been very nice to have while going through the book.
I will post further updates as I get farther into the book. I'm not much of a PL/SQL developer, but I will do my best as I get into the development chapters. In the meantime, I recommend checking out the sample that is linked above.
Wednesday, January 27, 2010
User Group Leaders' Summit Day 1 - #oraclesun
I was fortunate enough to be invited to the Oracle User Group Leaders' Summit at Oracle HQ in Redwood City this week. Once the Oracle-Sun deal was given the go ahead by the EC last week, we were invited to sit in live for the big announcement. My day started early after about five hours of sleep because my flight was late getting in last night. I met some of my fellow IOUG Board members for breakfast and ran into many others including Ronan Miles, UKOUG Chairman, who went to Brussels with IOUG President, Ian Abramson, to give the User Group position on the acquisition. Also ran into ODTUG President, Mike Riley, Quest President, Sue Shaw, and OAUG President, Ray Payne. That sounds like a lot of name dropping, and it is!
From breakfast, we walked over to the Oracle Conference Center and were ushered into the Auditorium. You can check out all my tweets from the session, but to sum it up, they are pushing hard on the integrated solution, complete platform for applications. We heard from Charles Phillips, who hoped we had a less eventful week than he has... Charles also talked about hiring 2000 people and if the current US Administration is looking for a jobs program, "her we are!" I won't go into politics here, but these are the kind of of things that should be focused on and learned from.
John Fowler, who is now EVP of Hardware Technologies at Oracle discussed the investments in Solaris and Sun hardware moving forward. Mike Splain, who seems to be the real techie, outlined the accelerated road maps for UltraSPARC and SPARC64. Oracle is putting more money into these platforms and they are hiring here too! My question continues to be how many people are getting laid off at the same time that they are hiring all these new positions. I'm sure the layoffs will get some money to walk away, but are they really just shifting positions? That hasn't been asked or answered yet.
The key theme continues to be seamless integrations between existing Oracle technologies and the newly acquired server, storage, and operating system. Supposedly all this testing has been done for many of the Oracle applications. Can they get these all working as they are advertising with flash technologies, new storage options, and newer Sun hardware and Solaris all remains to be seen. They keep selling the fact that they are being built and integrated together and if you have issues, you only have one support organization to deal with.
I'm still in the Oracle/Sun session and I'm sure there will be further information later. I also get a tour of the Usability Lab this afternoon which should be pretty interesting. Tomorrow starts the actual meetings including my first OpenWorld 10 planning meeting. I really wish we could finish one conference before starting planning for the next. Speaking of Conferences, don't forget about Collaborate 10! There will be sessions covering some of the new Sun technologies!
From breakfast, we walked over to the Oracle Conference Center and were ushered into the Auditorium. You can check out all my tweets from the session, but to sum it up, they are pushing hard on the integrated solution, complete platform for applications. We heard from Charles Phillips, who hoped we had a less eventful week than he has... Charles also talked about hiring 2000 people and if the current US Administration is looking for a jobs program, "her we are!" I won't go into politics here, but these are the kind of of things that should be focused on and learned from.
John Fowler, who is now EVP of Hardware Technologies at Oracle discussed the investments in Solaris and Sun hardware moving forward. Mike Splain, who seems to be the real techie, outlined the accelerated road maps for UltraSPARC and SPARC64. Oracle is putting more money into these platforms and they are hiring here too! My question continues to be how many people are getting laid off at the same time that they are hiring all these new positions. I'm sure the layoffs will get some money to walk away, but are they really just shifting positions? That hasn't been asked or answered yet.
The key theme continues to be seamless integrations between existing Oracle technologies and the newly acquired server, storage, and operating system. Supposedly all this testing has been done for many of the Oracle applications. Can they get these all working as they are advertising with flash technologies, new storage options, and newer Sun hardware and Solaris all remains to be seen. They keep selling the fact that they are being built and integrated together and if you have issues, you only have one support organization to deal with.
I'm still in the Oracle/Sun session and I'm sure there will be further information later. I also get a tour of the Usability Lab this afternoon which should be pretty interesting. Tomorrow starts the actual meetings including my first OpenWorld 10 planning meeting. I really wish we could finish one conference before starting planning for the next. Speaking of Conferences, don't forget about Collaborate 10! There will be sessions covering some of the new Sun technologies!
Wednesday, January 20, 2010
Stay Competitive: Join Me at the COLLABORATE 10 – IOUG Forum
I invite you to join me at COLLABORATE 10 – IOUG Forum, April 18-22, 2010 in Las Vegas, NV. As a member of the IOUG Board of Directors, I am excited to let you know about the educational content that the conference committee has created for the Forum.
You will have the opportunity to attend educational sessions in all of the tracks offered at COLLABORATE, including: Application Strategy and Services, Infrastructure Technologies BI/Data Warehousing/EPM, Database, Development, Governance Risk and Compliance, Master Data Management/Application Integration and Professional Development.
The IOUG Forum does not only offer top-notch educational sessions that cover a variety of topics, but it’s also a great place to network with other Oracle professionals. We are offering many exciting networking opportunities this year. Whether it’s a technical talk during a Birds of a Feather meeting or an informal conversation at the IOUG Welcome Event, the IOUG Forum is the place to meet contacts who can share solutions to your most pressing business questions.
The IOUG will not be the only user group present at the event. For the fifth consecutive year, the Oracle Applications Users Group (OAUG) and Quest International Users Group (Quest) will be co-hosting this event, which means your registration through the IOUG also allows you to attend education sessions offered by these groups.
The main reason I attend is for the networking opportunities with my peers. In all the years I have been attending IOUG events, the opportunity to discuss current technologies and best practices has been the best way I have found to keep myself current. The technical content is also amazing. These are people that are using the technology on a daily basis.
This isn’t some huge vendor show full of sales presentations…it’s presented by users like you and me. If you have been to IOUG events before, you know that we feature real-world education and best practices. There is far too much content to completely list, but you can visit the IOUG Forum Web site to view the most up-to-date education details.
Make sure to register to attend and join me at the IOUG Forum! Please visit the IOUG Web site for further details and to register. Be sure to register by March 18 to save $400 USD off the onsite rate, and choose priority code PC03 during the demographic section (step 4) of the registration process and gain access to a number of IOUG-exclusive offerings. Help me spread the word and send this information to your colleagues and friends who are Oracle users.
Please contact me or the IOUG headquarters at ioug@ioug.org if you have any questions.
I hope to see you in Vegas!
You will have the opportunity to attend educational sessions in all of the tracks offered at COLLABORATE, including: Application Strategy and Services, Infrastructure Technologies BI/Data Warehousing/EPM, Database, Development, Governance Risk and Compliance, Master Data Management/Application Integration and Professional Development.
The IOUG Forum does not only offer top-notch educational sessions that cover a variety of topics, but it’s also a great place to network with other Oracle professionals. We are offering many exciting networking opportunities this year. Whether it’s a technical talk during a Birds of a Feather meeting or an informal conversation at the IOUG Welcome Event, the IOUG Forum is the place to meet contacts who can share solutions to your most pressing business questions.
The IOUG will not be the only user group present at the event. For the fifth consecutive year, the Oracle Applications Users Group (OAUG) and Quest International Users Group (Quest) will be co-hosting this event, which means your registration through the IOUG also allows you to attend education sessions offered by these groups.
The main reason I attend is for the networking opportunities with my peers. In all the years I have been attending IOUG events, the opportunity to discuss current technologies and best practices has been the best way I have found to keep myself current. The technical content is also amazing. These are people that are using the technology on a daily basis.
This isn’t some huge vendor show full of sales presentations…it’s presented by users like you and me. If you have been to IOUG events before, you know that we feature real-world education and best practices. There is far too much content to completely list, but you can visit the IOUG Forum Web site to view the most up-to-date education details.
Make sure to register to attend and join me at the IOUG Forum! Please visit the IOUG Web site for further details and to register. Be sure to register by March 18 to save $400 USD off the onsite rate, and choose priority code PC03 during the demographic section (step 4) of the registration process and gain access to a number of IOUG-exclusive offerings. Help me spread the word and send this information to your colleagues and friends who are Oracle users.
Please contact me or the IOUG headquarters at ioug@ioug.org if you have any questions.
I hope to see you in Vegas!
Wednesday, December 30, 2009
Three Recoveries, Two Refreshes, One Delete
I feel like it's been a month since my last entry. Oh yeah, that's because it has been. I continue to be amazed with how fast time flies the older I get. Kids are supposed to make you feel younger and as they grow up, I just feel older!
The title of this post describes a full day I had a few weeks ago with a vendor application. Another title could have been "Support That Cannot Be Reached is No Support At All." The application in question has a relatively small database, but a couple hundred users. We actually converted to this centralized solution for the application about six months ago and the business owner has been converting locations two or three per week since.
I came into work to a my voice mail light... never a good sign. Apparently, they attempted to convert a large site the night before and had out grown the tablespace. "No biggie," I said and proceeded to increase the amount of allocated space. "We still can't get into the application and we cannot access any of the sites." This was very peculiar because this application has a different schema for each site and the only common information is stored in a "root" schema that manages some information on which databases are available. The user attempted to remove the affected schema from the application and I dropped it from the database, but this still did not allow the users into the application.
I'm not the most experienced person in the world, but I have dealt with quite a few applications during my career. I have never seen an application that just stopped working because of running out of space and especially not when more space was allocated. We contacted the vendor's support only to find out that their most experienced support analyst was out of town at a conference. The person we did get on the phone helped us check several things in the root schema, but couldn't seem to find anything that looked out of sync. The vendor suggested that we recover the database to prior to the load of the new schema. Knowing that the recovery wouldn't be a problem, I did have to warn them that since they had loaded the schema over twelve hours prior. This of course brings up other issues about notifications, etc, but that is a discussion for another blog posting.
Once we made the decision to go ahead and do the recovery, I looked at the dba_users view to find out when the affected schema had been created. Being the good DBA that I am, I copied the current data files to a backup location and started my RMAN recovery. Everything went perfectly and, as expected, the database came back up without the new schema. I called the application owner and told him to check it out. "It still doesn't work..." CRAP!!!!
This is the point when I start to wonder what is really going on here. It is also the point when the owner mentions that he actually loaded two locations the night before, not just the one. So, that would be recovery number two. I again figured out when the original schema was loaded and recovered to that earlier time. This time, when the owner tested the application, it worked. Okay, so at least we know which location caused the issue.
Users are allowed back into the application at this point and we plan for moving forward. The application owner says that he had tested that site in the test environment, but that he would do it again. When he tested it the second time, he found an error with the source data. Once the data was fixed, the load into test was successful. Somehow, I was talked into trying the load into production again since it worked in the test environment. I should have known better because as soon as he reloaded the site production was broken again. And that would be the explanation for recovery number three...
After the final recovery, we went ahead and copied the most current production data into the test environment. That would be refresh number one and when the data was loaded into the refreshed test environment, it broke it. At least we now had somewhere to try to figure out the issues. One more refresh and we waited to hear from the support person that new the most about the application. This is the part of the whole story that frustrated me the most. Once we had the right support person available, he was able to look and see that there was a data issue with one of the tables in the root schema. Deleted one row after loading the bad data and everything was fine. They deleted the row from the source data, reloaded into test, and everything worked. Being gun shy, we waited until the weekend to load the data into production just in case, but everything worked as expected, and he was able to load both sites without issue.
I learned a couple lessons throughout this ordeal. First, I had a gut feel that there was a data issue and not the tablespace issue that caused the application to not function. Unfortunately, I couldn't find any way to prove that and therefore had to go with the recovery that was being suggested by the vendor. This also leads to making sure the vendor has people available when needed to support issues. Of course, I didn't have anything to do about this because the vendor was chosen by the business. The other "stick to your guns" moment I had was when they wanted to load the data into production again after we thought it had been fixed. I should have insisted on a test refresh at that time to make sure we wouldn't break production again.
On the other hand, there were several good things that came out of this. My recoveries worked pretty well and I proved that I could go back to a point in time if necessary. The even better part of this was that the original data load was before the backup so we had to go back two nights and roll forward. I used to be a scripted backup and recovery person that could take or leave RMAN. After all these recoveries, to multiple points in time, I am an RMAN fan.
Well, I hope everyone had a great Christmas season, or whatever holiday you celebrated. Have a safe, happy, and prosperous new year. More posts soon. I really hope to get to this more often than I have recently. Also, make sure to look for information on Collaborate 10! See you in Vegas!
The title of this post describes a full day I had a few weeks ago with a vendor application. Another title could have been "Support That Cannot Be Reached is No Support At All." The application in question has a relatively small database, but a couple hundred users. We actually converted to this centralized solution for the application about six months ago and the business owner has been converting locations two or three per week since.
I came into work to a my voice mail light... never a good sign. Apparently, they attempted to convert a large site the night before and had out grown the tablespace. "No biggie," I said and proceeded to increase the amount of allocated space. "We still can't get into the application and we cannot access any of the sites." This was very peculiar because this application has a different schema for each site and the only common information is stored in a "root" schema that manages some information on which databases are available. The user attempted to remove the affected schema from the application and I dropped it from the database, but this still did not allow the users into the application.
I'm not the most experienced person in the world, but I have dealt with quite a few applications during my career. I have never seen an application that just stopped working because of running out of space and especially not when more space was allocated. We contacted the vendor's support only to find out that their most experienced support analyst was out of town at a conference. The person we did get on the phone helped us check several things in the root schema, but couldn't seem to find anything that looked out of sync. The vendor suggested that we recover the database to prior to the load of the new schema. Knowing that the recovery wouldn't be a problem, I did have to warn them that since they had loaded the schema over twelve hours prior. This of course brings up other issues about notifications, etc, but that is a discussion for another blog posting.
Once we made the decision to go ahead and do the recovery, I looked at the dba_users view to find out when the affected schema had been created. Being the good DBA that I am, I copied the current data files to a backup location and started my RMAN recovery. Everything went perfectly and, as expected, the database came back up without the new schema. I called the application owner and told him to check it out. "It still doesn't work..." CRAP!!!!
This is the point when I start to wonder what is really going on here. It is also the point when the owner mentions that he actually loaded two locations the night before, not just the one. So, that would be recovery number two. I again figured out when the original schema was loaded and recovered to that earlier time. This time, when the owner tested the application, it worked. Okay, so at least we know which location caused the issue.
Users are allowed back into the application at this point and we plan for moving forward. The application owner says that he had tested that site in the test environment, but that he would do it again. When he tested it the second time, he found an error with the source data. Once the data was fixed, the load into test was successful. Somehow, I was talked into trying the load into production again since it worked in the test environment. I should have known better because as soon as he reloaded the site production was broken again. And that would be the explanation for recovery number three...
After the final recovery, we went ahead and copied the most current production data into the test environment. That would be refresh number one and when the data was loaded into the refreshed test environment, it broke it. At least we now had somewhere to try to figure out the issues. One more refresh and we waited to hear from the support person that new the most about the application. This is the part of the whole story that frustrated me the most. Once we had the right support person available, he was able to look and see that there was a data issue with one of the tables in the root schema. Deleted one row after loading the bad data and everything was fine. They deleted the row from the source data, reloaded into test, and everything worked. Being gun shy, we waited until the weekend to load the data into production just in case, but everything worked as expected, and he was able to load both sites without issue.
I learned a couple lessons throughout this ordeal. First, I had a gut feel that there was a data issue and not the tablespace issue that caused the application to not function. Unfortunately, I couldn't find any way to prove that and therefore had to go with the recovery that was being suggested by the vendor. This also leads to making sure the vendor has people available when needed to support issues. Of course, I didn't have anything to do about this because the vendor was chosen by the business. The other "stick to your guns" moment I had was when they wanted to load the data into production again after we thought it had been fixed. I should have insisted on a test refresh at that time to make sure we wouldn't break production again.
On the other hand, there were several good things that came out of this. My recoveries worked pretty well and I proved that I could go back to a point in time if necessary. The even better part of this was that the original data load was before the backup so we had to go back two nights and roll forward. I used to be a scripted backup and recovery person that could take or leave RMAN. After all these recoveries, to multiple points in time, I am an RMAN fan.
Well, I hope everyone had a great Christmas season, or whatever holiday you celebrated. Have a safe, happy, and prosperous new year. More posts soon. I really hope to get to this more often than I have recently. Also, make sure to look for information on Collaborate 10! See you in Vegas!
Subscribe to:
Posts (Atom)
