Sunday, November 6, 2011

Stress Testing Oracle ADF BC Applications - Passivation and Activation

When we are talking with customers about Oracle ADF performance, very often we can hear such question - "Hey, ADF works well when there are few concurrent users, how it behaves when there is more serious load?". In order to answer this question I will publish series of posts, where we will study different parameters for AM pool tuning and test AM pool configuration under stress loads. Today we will see how stable is AM pool passivation/activation mechanism.

Stress tests are implemented with JMeter. Check Chris Muir blog about JMeter configuration for ADF 11g. I'm using JMeter configuration file from Chris blog, however original file is updated with internal loops to repeat session requests. JMeter allows to run stress test with multiple parallel users, each users starts its own HTTP session. It is not enough for stress test, additionally I would like to repeat user action within the same HTTP session - its why internal loops are needed, these loops allow to iterate through the same requests multiple times.

We will run 3 test scenarios (hardware: 4 virtual processors, 4 GB RAM):

1. Optimistic (15 concurrent users, 300 transactions in 10 minutes)
2. Average (25 concurrent users, 5000 transactions in 10 minutes)
3. Pessimistic (80 concurrent users, 16000 transactions in 10 minutes)

Each scenario consists of these steps:

1. Session initialization
2. Open Departments page (local AM)
3. Perform data change and Commit operation. Repeat 100 times per user, with 3 seconds wait time
4. Open Employees page (region imported from ADF library)
5. Perform data change and Commit operation. Repeat 100 times per user, with 3 seconds wait time

JMeter configuration file - internal loop to repeat session requests:


Wait time in the loop - 3 seconds:


Download sample application for stress test - stresstest_v1.zip. Sample contains two applications - main and application library. Both applications implement two AM modules, pool settings for both AM's are configured to the same values. There are two pages inside main application, first page brings data from local AM, second from imported region and related AM:



Second page integrates ADF region from ADF library:


Stress test is designed to run such scenarion, where user clicks on Submit button from ADF UI, this button calls custom method from AM interface. Custom AM method is accessing VO row, by randomized value in range based on VO rowset size. Numeric attribute value is changed (incremented by 1) for every accessed row, transaction is committed or reverted back, depending on concurrent modification. Additionally we have one transient attribute, it is set to be passivated:


All 3 stress test scenarios (Optimistic, Average and Pessimistic) are executed with the same AM pool configuration (for both AM's) - based on rough estimate of approximately 20 concurrent users in the system:


- Initial Pool Size = 22 (number of AM instances created on first ever access, recommended to be 10% more than estimated number of concurrent users)
- Maximum Pool Size = 30 (number of maximum AM instances pool can create, recommended to be 20%-30% more than initial pool size)
- Referenced Pool Size = 20 (number of active AM instances, recommended to be the same as estimated number of concurrent users - to avoid frequent passivation/activation)
- Minimum Available Size = 0 (when set to 0, allows to release all AM instances after idle timeout, this helps to release reserved database connections as well. Is set to 0 for tests in stress environment, in your system you may set it to higher value, but less than referenced pool size)
- Maximum Available Size = 25 (maximum number of AM instances after pool clean-up)
- Idle Instance Timeout = 300 (AM instance is considered inactive after 5 minutes of inactivity, this if for stress test. In your system you would set it something close to Web session timeout - to prevent frequent passivation/activation)
- Pool Pooling Interval  = 120 (AM pool is cleaned every 2 minutes)

1. Optimistic (15 concurrent users, 300 transactions in 10 minutes)

This test is performed on first ever access. As you can see, 22 AM instances are created, as per initial pool size setting. However, soon (idle time 5 mins + pool cleaning interval 2 mins) 7 AM instances are removed - because we have only 15 online users. Passivation/activation never happens (because referenced pool is set to be for 20, and we have only 15).  MainModule statistics:


Same for HrModule module:


Very important, because Minimum Available Size = 0, after idle time entire pool is cleaned:


Same with DB connections, there is no need to wait for AM time to live, AM instances and DB connections can be released earlier - however this will trigger passivation. DB connection can be released before AM time to live, by tuning AM pool Minimum Available Size and AM Instance Timeout:


2. Average (25 concurrent users, 5000 transactions in 10 minutes)

This test with 25 online users is slightly above Referenced Pool Size = 20. We should experience passivation/activation behavior.

AM pool allocates 20 active instances for 25 users (as per Referenced Pool Size). Passivation/activation starts to take place, in order to support all 25 users (around 500 passivation/activation circles per minute):


Similar behavior for the second AM:


40 DB connections are used during peak time, when both AM's are active (20 + 20 active AM instances):


As you can see, even Maximum Pool Size = 30, active instance size is not exceeding Referenced Pool Size.

3. Pessimistic (80 concurrent users, 16000 transactions in 10 minutes)

With 80 users, AM pool starts to passivate/activate much more (as expected). There are 20 active AM's,  as per Referenced Pool Size and those 20 AM's need to serve all 80 users. There are lots of switch activity happening - around 1500 passivation/activation circles in 1 minute:


Runtime statistics for the second AM:


During strong stress test, we can see that we were using at some moments more than 40 (20 per each AM) database connections (as previous test). Means AM pool during heavy stress test may maintain more active AM instances than specified by Referenced Pool Size - but still, AM size will not exceed Maximum Pool Size setting (30 in our case):


Conclusion: AM Pool is strong enough to handle stress peaks with online users greatly exceeding number of AM Pool Size and and AM Referenced Pool Size. However, in your system you should try to maintain AM Pool sizing as close as possible to expected number of online users - this will help to avoid performance costly passivation/activation operations.

23 comments:

Chris Muir said...

As usual a useful post Andrejus, thank you.

Can I ask a question on the "AM Maximum Pool Size" setting. Up to now I thought that was a hard limit that when hit, the framework will throw errors. Is that your understanding?

An alternative to that last question, can you see any reason not to leave the Maximum Pool Size as the default 4096 value?

Also in the last Pessimistic scenario, against diagram 3c.png you comment "we can see that we were using at some moments more than 40 (20 per each AM) database connections". If that's the case, in diagrams relating to the Pessimistic scenario, shouldn't we see the Active Instances grow above 20 to reflect the growth in the pool size?

Cheers,

CM.

Andrej Baranovskij said...

Hi Chris,

"AM Maximum Pool Size" - based on stress test, this setting is not a hard limit. If "AM Maximum Pool Size" is set to smaller value, framework just executes slightly more passivation/activation, but still serves all users (however number of passivation/activation mostly depends on "Referenced Pool Size"). As per doc, "AM Maximum Pool Size" should be 30% more than expected concurrent number of users, it looks like there is no need to keep 4096 - at least is not giving any visible performance effect to keep "AM Maximum Pool Size" large. However I will continue tests on this.

At the same time, I don't see so far any reason not to leave "AM Maximum Pool Size" = 4096, unless AM Pool is wasting memory when this value is set to be large - will need to test memory consumption.

For Pessimistic scenario - yes, its what I was thinking as well. Why we don't see more than 20 Active Instances. I guess, during stress test conditions, AM Pool maintains some AM instances in transition state (moving from Active to Passive). May be its not able to switch AM instance fast enough, its why it consumes slightly more DB connections.

Thanks,
Andrejus

AnAverageJoe said...

Thanks Andrejus - will follow this with interest :)

Andrej Baranovskij said...

I was doing more tests, situation became more clear now.

- "AM Maximum Pool Size" is not hard limit. However, on very high load and when WLS is running out of free resources - AM pool expands and creates more active AM instances, then specified by Referenced AM Pool size. It keeps growing until "AM Maximum Pool Size" and then it stops serving new users. However, this happens only with really heavy stress test - usually AM pool tries to passivate/activate fast and stay in limits of Referenced AM Pool size - rarely growing higher and reaching Max size.

- We saw more DB connections on high load, because of internal AM connection for passivation/activation. I declared jbo.server.internal_connection for both AM's and pointed to separate dedicated Data Source - now number of consumed connections stays stable.

Andrejus

M.Ahmed said...

Hi Andrejus,
I want to know how to get the graphs that indicates the number of App Module instances loaded.

Andrej Baranovskij said...

Hi,

You should use Oracle Enterprise Manager to access these graphs.

Andrejus

Tsendee said...

Hi.Andrejus

How do you connect JMETER with ADF?Is it possible to use JMETER with ADF?Please tell me.

Andrej Baranovskij said...

Yes, read second paragraph of this post, where you will find link to Chris post :)

Andrejus

Ganbat Bayarbaatar said...

Hi, Andrejus!

I've read your posts about stress tests, AM Passivation/Activation, AM Pool Settings and doconnectionpooling=true. And Thank you for these useful information.

I'd like to ask you 2 things:
1. We have nested AMs under one RootAM. Is it enough to set these settings just to the RootAM? Or do we need to set it to all AMs.
2. We've encountering problems about max-user-connection. And would like to use doconnectionpooling=true (with disconnect_level=1). But when we set it like that, we get an java.lang.reflect.InvocationTargetException error. It throws when our App try to execute VOImpl method which locates under one of the child AM.

FYI, we use MySQL for db, JDEV 11.1.1.4.

Thanks,
Ganbat Bayarbaatar.

Andrej Baranovskij said...

Hi,

1. Yes, its enough to set tuning settings only on root AM

2. I didn't encountered this error, could you send me reproducible sample test case - would be interesting to check

Thanks,
Andrejus

Unknown said...

Hie

How do you actually run the below cases (I mean where you set these?)

We will run 3 test scenarios (hardware: 4 virtual processors, 4 GB RAM):

1. Optimistic (15 concurrent users, 300 transactions in 10 minutes)
2. Average (25 concurrent users, 5000 transactions in 10 minutes)
3. Pessimistic (80 concurrent users, 16000 transactions in 10 minutes)

Andrej Baranovskij said...

Hi Vik,

I set concurrent users and number of loops in JMeter. Then I calculate based on concurrent users and number of loops how many transactions it will execute (you can see this from JMeter log file as well). Time is approximate - it takes around 10 minutes for JMeter to execute each of the tests.

Andrejus

Jamal Shaaban said...

Hi Andrejus,
I have a question, does every page in adf application takes a connection from pool? if the system has 10 pages and the concurrent users is 10 then i must have 100 connection ? i'm asking also what to do if the same user closes his session and open new session, is he considered a new user?/

Andrej Baranovskij said...

No, in ADF - Application Module takes DB connection, not the page. To minimize DB connections usage, you can enable DB pooling for AM.

Regards,
Andrejus

Jamal Shaaban said...

Hi Andrejus,

I already enabled application module pooling. when i saw the number of taken connections when monitoring the server, each page i entered take a connection form the pool. could you please help me if i had to do another configurtions ?

regards,
jamal

Andrej Baranovskij said...

If every page takes new DB connection, this means application is developed badly. You should review code and fix errors. What errors exactly, hard to say, without reviewing your code.

Andrejus

Jamal Shaaban said...

Hi Andrejus,
the error was from me,In my test when i made a request to the page i removed the adf-ctrl-state parameter so it is considered as new session .The whole Application takes on connection only. sorry Again for misunderstanding

best regards,
Jamal

Anonymous said...

Hi,
I use jdev 11.1.1.7.
I have 700 concurrent user and
I need to call several iReport with JDBC connection .
Now how should i config my appModule for best performance?

oceans said...

Hi,
I would like to know if the ADF framework can do a lazy passivation for performance improvement. Say I want the passivation to happen at the end of every request (fail over true) but only after the AM instance is released to the pool. So the passivation will not be a part of my request cycle.
From the Doc i saw a similar flag PASSIVATE_DEFER_FLAG , but not sure how it can be used for this usecase.
I am using jdev 11.1.1.7

Andrej Baranovskij said...

Good question, I will check it and follow up.

Regards,
Andrejus

oceans said...

Hi Andrejus,
Got anything on Lazy passivation stuff ?

Andrej Baranovskij said...

Not so fast, too busy with my work in the projects. This is on my todo list.

Andrejus

oceans said...

Sure, thank you for the effort.