Child pages
  • 05_01_2009 U-M W09 Review & EvalSys Team Meeting (Sakai Bridge 001)
Skip to end of metadata
Go to start of metadata

U-M Winter 09 Review and EvalSys Team Meeting

Mon May 11, 2009

1. Background

  • What happened at U-M

2. Brief overview of initial findings

  • CTools/Sakai issues:
    • Presence
      • It uses the DB for communication in the cluster but it shouldn't and can be re-implemented to avoid using the db.
      • It significantly increases demand for connections from the db pool on each appserver
    • Runaway WebDAV sessions
    • Database connection pooling
    • Very heavy load of realm queries
    • Very large and growing system tables
    • Email Digest processing
  • TQ issues:
    • U-M branch:
      • Instructor widget
      • UMIAC queries
      • Inefficient email processing leading to spiky load
      • Lack of throttle mechanism
    • Trunk:
      • Specific HQL calls

3. Next Steps

-----

Meeting notes (by JL):

5/11 9am eval con call
summary of 4/15,20,23 events

presence - http requests, faily linear
smarq - contributing factor, some percentage of overall load
impl jms significant change
update every 2-3 min instead of 30 sec.

runaway webdav
login, authentication are expensive
2 jiras on agenda page - from smarq

db connection pooling
new db hardware 64Gb ram, faster hw

email digest processign

tq issues
added in count widget, preview of eval
counting # students in class is very expensive query

admin, currently, superuser overseer, manage any details of evals for someone else
admin on summary and my evals, sees everything - perf problem
cambridge has only a couple 100. If thousands, wouldn't work
needs paging

authz checks - recent change to eval - don't go to authz, use faster check in eval tables
wait until eval starts and update enrollment then, and don't update again, use that record
instead of going to authz each time.
another reason - allows more flexibility tohave users participating who are not actually in enrollment

umiac queries
thread bug
email processing time
email delivery

lack of throttle mechanism

hibernate, some of the hql calls
vasily - evaldaoimplementation class - couple queries there hql statements - 1=1 means a table scan
vasily - is it by intention? az - no
method getevaluationsbyevalgroups
other places distinct used - getqueuedemaillock, getitemgroups - that also may cause full table scan

smarq - optimization but not orders of mag, sometime just need to add more app servers, db
connection pooling - if load is at limit, then you just need more capacity
adi - contention on write to same block - spikes, not in general. we'll be adding more connectgions,but its not the pool, thats a symptom
smarq - ran at 50 in pool, peak up to 600
10/appserver iswhat it is 90% of time
most sig issue - stagger emails to students so not getting a spike
smarq - need to spec system to survive peaks 2-3x times typical heavy load

az - they run 250 connections in pool. not high right now, 150 connections active currently
um - see typically 8 total active/second out of 15-20 per machine
ax - running 5.1.5 mysql

eval_answercontention - vasily see anything there?
v -could be table scan related to select, but insert seemed to be clean

  • No labels